Input preparation¶

The acud simulation input data is specified through a set of text files which can be grouped as follows:

configuration yaml files
content csv files

Configuration yaml files¶

Referred to in short as yml files, these are two YAML-format files:

run configuration yml file, and
model configuration yml file

which jointly define the model outline.

The model yml file specifies the more “structural” data whereas the run yml file contains information related to the analysis to be performed (e.g. which solver to use or the time horizon to be considered). The split of data between run and model configuration data is somewhat arbitrary, the purpose of this split being to ease the reuse of information contained in the model configuration file. In this sense, there is a certain hierarchy between the two config files, with the run file at the top level. Indeed, the run yml file can be thought of as the trigger file: it is the only file that needs to be directly specified when invoking an acud command (the model configuration file is invoked internally through the acud code using the model entry of the run configuration file).

The yml files themselves do store some of the values applied in the acud simulation, e.g. discount rate or VOLL, but most of their content is made up of references to other files (mostly csv files) where the data is actually stored.

yml files store nested mappings of key-value pairs, where values can be either single values, lists of values or further key-value pairs. Mappings are nested through indentation, e.g.

timeseries:
    generator:
        derating:
            format: ymdp
            files:
               COAL: input/derating_COAL.csv

In the example above the key-value pair COAL: input/derating_COAL.csv is a value to the files key which in turn together with the format key-value pair is a value to the derating key, and so on.

IMPORTANT: use space NOT tab for indentation to ensure correct yaml file parsing.

In these notes we refer to the most inner entry of nesting mappings using a “flattened” dot-based notation, e.g. COAL files entry in the example above would be represented like this

timeseries.generator.derating.files.COAL: input/derating_COAL.csv

Within a given map nesting level, the order in which the entries are written in the configuration files is not relevant but it is probably best to follow a certain convention about it.

Below we provide a commented version of the run configuration file. As YAML allows comments (after #), the text below is valid config file content.

name: basecase_run  # run name, can be different from model name
model: model.yml  # name of (and path to) file with model configuration data
default: default.yml # name of (and path to) file with parameter default values

time:
    startperiod: 2012-03-01 00:00  # use this datetime format
    endperiod: 2012-03-31 23:00 # last period included in modelling horizon
    stepsize: 24 # number of hours in optimization step
    blocks: 3 # placeholder, currently not in use

output:
    format: csv  # only format currently supported
    path: solution  # results destination, will be created if it doesn't exist
    drypath: dry # dry run output destination, will be created if it doesn't exist

settings:
    mode: quantity  # placeholder, currently not in use
    phase: MT  # LT, MT or ST
    solver: glpk # CBC or glpk
    cat: Integer # Linear or Integer solution for integer variables

Similarly, a commented version of a sample model configuration file follows.

name: basecase   # model name

components:
    bus:
        file: input/buses.csv
        referencenode: GB # if no bus specified for generators, assume they are connected to this bus
    branch:
        file: input/branches.csv
    generator:
        file: input/gens.csv
    fuel: # can be empty, but needs to be included for the fuel timeseries input specified below to be processed

timeseries:  # this information further qualifies the component instances created based on the components entries above
    generator:
        derating:  # this entry could also be a time invariant parameter, in which case there would  be no need for a timeseries entry
            format: ymdp
            files: # as the shape is ymdp, each file contains information referring to a single component instance
               COAL: input/derating_COAL.csv
               OCGT: input/derating_OCGT.csv
               CCGT: input/derating_CCGT.csv
               OIL: input/derating_OIL.csv
               NUCLEAR: input/derating_NUCLEAR.csv
               OTHER: input/derating_OTHER.csv
               WIND: input/derating_WIND.csv
               PS: input/derating_PS.csv

        units:
            format: dtname
            datetimeformat: '%d/%m/%Y' # as in input data file, the internal format is different
            files:  # can be a list of multiple files which will overwrite the configuration derived from the component table entries
               - input/units_WIND.csv
    fuel:
        price:
            format: dtname
            datetimeformat: '%d/%m/%Y'
            files:
               - input/fuels.csv

    bus:
        load:
            format: ymdp
            files:
               ES: input/load.csv

scalars:   # these are values that are not indexed neither by datetime nor by component instance name
    discountrate: 0
    voll: 5000
    pricedumpenergy: 5000
    loadescalator: 1
    branchescalator: 1

yaml templates¶

Configuration yaml files are a key ingredient of an acud input dataset. They contain a number of fields grouped by sections and subsections, and can be easily created and updated using the simplest of text editors. However, some complication arise when dealing with configuration files:

they must abide to the [YAML file format](http://yaml.org/) e.g. regarding indentation
they involve a long and nested list of required and optional fields.

YAML file templates tackle these two complications. They are well formatted (i.e correct indentation and markup) and contain placeholders for all available fields -whether optional or required- with descriptive comments attached.

Let RUN_CONFIG be an existing yaml file at the case folder location where you want to create new yaml files (note that for the purpose of automatically creating a template-based yaml file, RUN_CONFIG does not need to be a properly built acud run configuration file).

To create a draft run config file from an acud template, use the following command:

ad dry RUN_CONFIG --run

Similarly, for a model or a default file:

ad dry RUN_CONFIG --model
ad dry RUN_CONFIG --default

You can also create multiple templates simultaneously:

ad dry RUN_CONFIG --run --model --default
ad dry RUN_CONFIG --run --model
ad dry RUN_CONFIG --run --default

and so on.

This command will create a file called, respectively, run.yml, model.yml and default.yml at the location where RUN_CONFIG is stored. If there is already a file called run.yml, model.yml or default.yml at the destination folder it will not be overwritten. Instead, the template-based yaml files will be renamed as run(1).yml, run(2).yml and so on. Extracts of the run, model and default templates look, respectively, like this:

# RUN CONFIGURATION

name: <run name>
model: <model yaml file>       # Relative path to model configuration yaml file
#default: <default yaml file>   # Relative path to user-defined default yaml file that overrides built-in defaults

time:
    startperiod: <YYYY-MM-DD hh:mm>                # Starting hour of simulation horizon
    endperiod: <YYYY-MM-DD hh:mm>                  # Last hour of simulation horizon
#    stepsize: [int]                                # Number of hours in optimization step
#    blocks: [int]                                  # Number of blocks applied in load duration curve approximation
#    snapshots:
#        file: <snapshots csv filename>             # Full filename to snapshots specification
#        datetimeformat: <Python datetime format>   # Pattern used when parsing datetime field, see http://strftime.org/ for syntax reference

output:
    path: <solution folder>                   # will be created if it doesn't exist
    drypath: <dry run output folder>          # will be created if it doesn't exist
    xtrapath: <extrapolated solution folder>  # will be created if it doesn't exist
#    format: [csv]                             # File format of reported solution
#    scenario: [scenario name]                 # Label for data handling
#    out_lp:  [0 or 1]                         # Switch to output text file with problem formulation in CPLEX LP format
#    out_mps:  [0 or 1]                        # Switch to output text file with problem formulation in MPS format

# MODEL CONFIGURATION

name: <model name>

components:
    bus:
        file: <filepath>                          # Table file location
        referencenode: <string>                   # System wide energy price is set to nodal price at bus reference

    generator:
        file: <filepath>                          # Table file location
        datetimeformat: <Python datetime format>  # Pattern used when parsing date from and date to field, see http://strftime.org/ for syntax reference

#    branch:
#        file: <filepath>                          # Table file location
#
#    fuel:
#        file: <filepath>                          # Table file location

scalars:
#    discountrate: [float]              # Yearly rate to discount cash flows in objective function of capacity expansion problem
#    thermalfirm: [0 <= float <= 1]     # Firmness factor of thermal generating units in capacity expansion problem

# SOLVE OPTIONS
solve:
    mode: quantity  # price, quantity
    phase: MT  # LT, MT
    solver: coin  # coin, glpk, xpress, gurobi
    cat: Integer # Integer, Binary or Continuous
    scenario: Base
    allow_use: 1  # allow unserved energy
    allow_dump: 1  # allow dump energy
    include_ramps: 1  # consider max ramp up limits
    include_msl: 1  # consider min stable level
    include_no_load_costs: 1 # consider no load costs
    include_start_costs: 1  # consider start costs
    epgap: 0.0001  # mipgap tolerance termination
    sec: 7200

# TIME OPTIONS
time:
    stepsize: 24
    blocks: 3


# OUTPUT OPTIONS
output:
    verbose: 0 # amount of progress info printed
    out_csv: 1 # write csv files

Composing the yaml file is then just a matter of editing the placeholders in the template and saving changes. Note that only the default.yml is ready to use as provided; in this case the file does not include placeholders but rather built-in default values. The run.yml and model.yml need further editing by replacing the placeholder included in the template following the field code described below:

<item> denotes that user-defined input is required, i.e. acud provides no default value for this entry.
[item] denotes that a default value is available for this entry, i.e. if you do not provide a value for this entry then acud will apply a default value during the simulation.
Commented out entries are optional entries; when uncommented during the template modification, these entries need to be further edited by replacing <item> or [item] placeholders with the appropriate value.

Content csv files¶

Content files are a variable number of csv files containing the bulk of the input data. There are two types of csv files:

table files
time series files

Non-data rows can also be included in csv files, regardless of their type. Rows starting with a # character will be ignored by the acud computations.

IMPORTANT: the acud code is case sensitive so upper and lower cases of component instance and parameter names need to match.

Table files¶

Table files lists the instances of the different components included in the model. There should be one table file per component type contained in the model. The first columns shows the name of the instance, the rest of the columns store various parameters, which will vary depending on the component type.

Component instance parameters can be either numerical, denoting measures of quantifiable metrics, or categorical, representing relationships. To illustrate, generators have numerical parameters such as heat rate or max capacity, and categorical parameters such as bus (to which it connects) or fuel (burned).

When acud reads the input data it starts with the table files. Reading through these files it creates the different component instances that comprise the model, populating their parameters either with provided or default values.

acud input processing then moves on to time series files which further qualify the information provided through the table files.

Time series files¶

Time series files come in two (for now) shapes:

dtname
ymdp

You need to specify the shape of the time series files through the format entry found in the model configuration yml file, e.g. like this:

bus.load.format: ymdp

or

fuel.price.format: dtname

In csv files with dtname shape the first column contains datetime values and the header for this column is “datetime”. The rest of the columns to the right store parameter values, with each column containing the values of a single component instance. The names of these component instance are stored as headers of the value columns. To avoid faulty datetime parsing it is best to explicitly specify the datetime format through the entry:

timeseries.<component type>.<parameter>.datetimeformat

csv files with ymdp shape store year, month and day values in the left columns. Each of the remaining columns contain the parameter values, with each column storing values for the same period (of different days). Periods are shown as column headers. csv files with ymdp shape do not include any component instance name information, which means that each csv files contains timeseries data corresponding to a single component instance.

In principle most if not all parameters can be made to hold time series. However the mathematical optimisation problem formulation assumes that only a given subset of parameters are time dependent, with all other parameters being time invariant.

The time dependent parameters in the optimization problem formulation are as follows.

For generators:

max capacity

max build

cincr (incremental costs)

units

derating

bid

bid escalator

For buses:

load

For fuels:

price

All other parameters not listed above will be handled as time invariant in the optimization problem formulation. When a time series is specified in the simulation input data for a time invariant parameter, the value of the time series at the start of the modelling horizon will be assigned to such parameter.