PJM Invoice Process

Overview

The PJM Invoice process retrieves the weekly and monthly invoices for each sub-account used by Active
Power for trading on the PJM market. The data retrieved is disseminated to the PJM_Accounting
database and used in reporting.

- Retrieving the reports to process

-

1. Run pre-processing

This gets the Available reports - the code makes a call to the PJM website to pull down the
report availability file. This file contains the available reports and the date that the report is
available. The fields in the file are Report_Name, Start_Date, End_Date, Version,
Version_TimeStamp and Published_Status. The file is loaded to the table
“Working_AvailableReports” table. Prior to loading, the table is cleared by using the truncate
table command in the “AvailableReports_PreProcessing” stored proc.

Once the table is loaded, the file is deleted and the post processing stored proc
“AvailableReports_PostProcessing” is run. This procedure does the following.

Inserts data from the InvoiceCalendar table to the ReportCalendar where the data does
not currently exist based on a 7 day “look back” period of the InvoiceCalendar. This data is
based on the Start and End dates (reporting period)

Inserts data from the Working_AvailableReport table to the AvailableReport based on

the data isn’t currently in the table and the criteria that the report status is “Available”

The application then loads the reports from the AvailableReports table to the ReportCalendar
table. This ReportCalendar table is the main table for all reports and the dates that the reports
are available for the application to download the report to the working directory.

This process ensures that we have all of the reports in the ReportCalendar regardless of what
the report is (i.e. Invoices, Settlements, etc.)

This proc could use some performance improvement

The downloaded ReportVersionAvailability report is then deleted.

The loading of the InvoiceCalendar table is handled by another process called:

PJMCalendarDownloader. The documentation for this is located on Magnus
D:\ET\_Infrastructure\Process Documentation\PJM\PJM Calendar Downloader.docx
2. Getting the reports we need for processing

The stored proc “PJMReports_GetToday” is run to get any report that is available.

This is accomplished by inserting any information from the ReportCalendar (reports we need)
with the credential information for each report. A report may, and typically will be, used by each
account (i.e LANTRE, LANTS1, LANTS2 and LANTS3). This proc joins the reports to the
credentials/accounts and inserts them into the table “ AccountReportRequest”.

The proc then returns all the data along with the credentials for the accounts to the application
so that they can be processed.

The data is returned as a list of the class of “ReportRequest” this class contains the values for
each report request. These values contain properties that are passed to each of the processes to
be handled by called method. An example is when the DownloadReports method is called, the
list of ReportRequest is sent, the download process build the list of HTTPAttributes to then pass
to the method that will actually handle the downloading of the file.

3. Downloading the reports

The list of HTTPAttributes is passed to the HTTPFileDownloader which is an assembly that is set
as a reference. This assembly is used by other applications and handles the downloading of files
from the HTTP Client or a SFTP server. The PJM reports are an “on demand” request in that we
build a url to pass to the PJM API located on the PJM website. This call returns a file of the type
we specify in the calling url (typically .XML and .PDF for this application) and the file is saved to
the location specified in the HTTPAttribute object that was created by the ReportRequest object
that was created from the settings in the database.

4. Insertion of data to the database

To insert the data to the database, we use a method called “InsertDataIntoDatabase” (seems
kind of intuitive huh?) this method takes a IReportLoadingTablesHandler object, as well as the
source path (string), fileExtention (string) and searchCharacters (string). This method
instantiates a PJMReportFileToDataSet object. This object is responsible for taking the
parameters of the sourcePath and the searchCharacters. These two parameters are responsible
for informing the class where the files are located and what the files should contain to use for
processing. The result is a data set that is then investigated for the underlying data tables. Each
of the data tables is then passed to the static method “LoadData” of the
APICoreDataSQLBulkLoad class. This method attempts to take the data table along with the
database connection to insert the table to the database table with the same name to the
database specified in the connection string. Once the data has been loaded to the working
tables, a call to the stored procedure “” is invoked and the database side transformation
processing begins. When the return is successful, then processing continues.

5. Processing the files

 After the file(s) have had their data inserted in to the database, each file is processed in the
“ProcessFiles” method, which takes several parameters in. these Parameters include the List of
ReportRequest objects as well as the source and target path (directories), file extension of the
files that we are interested in and optional “search tail”. This method creates a list of the report
requests that contain the file extension that was passed, so not all of the report requests that
were passed in will be processed, only the requests that match the file extension that was
passed. Each of the newly created list of requests are used to create the search and target path
for passing to the FileManipulation class. The FileManipulation class is a class that handles the
distributing of the files based on the date mask passed on. An example is a file name sucha as
the following “20191222_Somename.xml”. The class will place that file in a directory with the
following path” <target directory>/2019/12/22 this will organize the file based on the date
contained in the file name. This class was written as it is a familiar way of organizing files of this
type.

6. Completion of the process

Once the data from the files have been inserted to the working tables, post file processing has
completed and the files have been moved and zipped if needed, the processing is complete.

The application updates the status of the report request to the database. If everything was
successful, then the flag IsSuccessful will be set to true, if not, the IsSuccessful flag will remain false
and will be processed the next time the application is run.