When opening the Intuitive Dashboard, there’s no data present in the components and/or an error is displayed within the component.
Altering the Datafeed Schedule
To narrow down the cause of the error, we'll take a look at the process involved to import data from PaperCut into the Intuitive dashboard. By better understanding the data loading process we will be able to indicate at which stage the import of the data has failed.
PaperCut / Data Integrations
The below diagram illustrates the process involved, so we'll start from the PaperCut MF application.
PaperCut is configured with one of several database types, we've included below a list of potential applications:
- MSSQL
- Apache Derby
- Oracle
As Intuitive Dashboards uses the 'Data Integrations' feature within PaperCut, it will export your PaperCut data into .csv form. This means that the database type your PaperCut installation is using will be supported, so long as the data is exported by this feature.
By default, PaperCut is configured to export the data during its maintenance window at 12:55 am each morning, but for more information surrounding the maintenance window, please consult the this article from the PaperCut knowledge base.
The first area for you to check is if PaperCut has exported your csv's for the Intuitive Dashboards to read. By default, this folder is located below, but may differ if this was reconfigured to an alternate path:
C:\Program Files\PaperCut MF\server\lib-ext\data-integrations\data\bi-exports
Within the above directory, you should see a folder structure similar to the below, with each year and month having it's own directory containing the exported csv's from PaperCut:
If this data is present, we'll move onto the next stage within this process.
If there is no data present, this can be due to several reasons. Firstly, you may not of reached your maintenance window if you have only just set up data integrations. Secondly, if you have already waited overnight, you can manually restart the PaperCut service which will mean downtime on your PaperCut installation. Thirdly, you can manually generate the csv's via the "Load Integrations" link in PaperCut under Reports, then Data Integrations.
ETL Service
The Intuitive ETL service is the next stage in the data loading process. The ETL service retrieves the individual csv files and collates them into a SQLite database where the data is aggregated and stored. This will act as the data source for the Intuitive application. The below diagram illustrates where we are in this process:
On the ETL service's first scan, it will run through every single csv in the aforementioned bi-exports folder, and then on its subsequent schedule at 2 am each day, it will scan this directory again and go through any incremental changes to this folder.
To check whether the ETL service has uploaded these records onto the database, you can find its log files via the below location:
C:\Intuitive_WorkFiles\Logfiles
The log files for the ETL service have a "Scheduler" prefix with a date stamp appended on the end. This timestamp format is configured as year, month and then day, i.e. for the 17th of November 2021, this will be "Scheduler.20211117.txt".
This is the next area for you to check, if you open up the first scheduler log file available, you can scan through the document for any errors during this process. You can then check the latest available scheduler file, to ensure it has subsequently imported the newest csv's available. If there are no errors present, then this data will have been added into the database.
For reference, we've included two examples of unsuccessful and successful ETL log files below.
This screenshot below is for an unsuccessful data load, you can see from the log messages that it's unable to load data for data from November 2020. The reason for this particular issue is that the updated data from PaperCut hasn't yet been exported, so the ETL service is expecting a file which hasn't yet been generated.
Below is a successful import, you can see it is inserting the csv files into the respective tables and it finishes with the "SLEEP" message. This message indicates that the ETL service will wait until its scheduled time to update, then it'll start this process of importing the data from the PaperCut csv's again.
Should you wish to change the time in which the ETL service starts, we've included a guide down below.
If not all of the data has been imported into the database from the csv files, you can force Intuitive to reload the data. To do so, delete the .properties file in the below location:
C:\Program Files\Intuitive Business Intelligence\Intuitive Dashboards\Tools\ETL Service\Initialised
Then restart the Intuitive ETL service. You can check on its status by checking the scheduler log files in the below location:
C:\Intuitive_WorkFiles\Logfiles
One reason none / not all of the data has been imported can be down to how soon the dashboard service is started after the ETL. It is important that the ETL fully runs before the dashboard service is started, otherwise it will create the feed files from a blank database.
Datafeeds
Following a successful run of the ETL service, the next stage of the process is for the Datafeeds to load the data into the Intuitive application. The diagram below shows where we are in the process.
The Intuitive Datafeeds load tables from the SQLite database into the Intuitive Dashboard Service where they are cached in memory as Datasets. The Datafeeds manage the data loading process and are scheduled to update at 4am each day.
As part of the data loading process the Intuitive Dashboard Service creates feed files which are stored on the server. The purpose of the feed files is to provide a cached copy of the data to the Intuitive Dashboard Service in case of a restart.
You can find the feed files in the location shown below. You can also see the individual feed files. There is a file for each datafeed configured so for a standard PaperCut installation you will see 4 files:
C:\Intuitive_WorkFiles\Feeds
Should there be no feed files present or the files present are very small, this is one indication that the data from the SQLite database hasn't imported properly. To ensure their size is correct, you can check the dashboards to ensure the data is going as far back as you'd expect.
To force the dashboard system to reload the data from the SQLite database you need to delete the feedfiles or alternatively you can refresh the feed by editing the datafeeds in the Datafeed Builder in the Intuitive application.
At this point, start / restart the Intuitive Dashboard service and check that the feed files are present.
Note: Allow a minute between stopping and starting the Intuitive Dashboard service.
By default, the datafeeds within the Intuitive Dashboard are configured to refresh the data at 4am each morning, but should you wish to reconfigure this time we've listed a guide below.
In addition, you can check the datafeeds have imported the data by going through the below process on the Intuitive Dashboard:
- Login to your dashboard
- Click the menu icon in the top right
- Hover over "Datafeed Builder" and select "Edit datafeed"
- Navigate to one of the datafeeds, i.e. "Departmental Statistics - 12 Months"
- Open the file and check there is data onscreen, below is an example from "Departmental Statistics - 12 Months.
You should now be able to view the data within your dashboard.
Permissions
If you are unable to open any of the dashboards, as shown below, your account may not have access to the available dashboards.
To check / edit the account permissions, please follow the below steps:
- Login to your dashboard as admin
- Go to the menu icon in the top right
, hover over "Admin" and select "Users" - Select the user on the left i.e.
- Click the Edit icon
- On the right hand side of the screen, there will be a hierarchy of objects that your user has access to.
- To give your user access, select the applicable checkbox, i.e. "Intuitive for Papercut V3.0"
- Save your changes
- Logout as admin via the menu in the top right, then log back in as the problematic user and check that you're now able to view dashboards, as shown below:
After reviewing this article and should you be unable to view the data within your dashboard, please contact support@intuitivebi.com
Altering the ETL Schedule
To alter when the ETL service picks up the new csv files from PaperCut, you need to make several alterations to the ETLService.config file. This can be found in the below directory:
C:\Program Files\Intuitive Business Intelligence\Intuitive Dashboards\Tools\ETL Service
We recommend you use a source code editor, such as Notepad++ to make these changes as we'll reference particular line numbers, since there is no global variable for the ETL service.
To ensure there are no permissions errors when saving this file, open your source code editor as administrator.
However, before making an changes, please ensure that you leave a gap between your PaperCut maintenance window, your new ETL time and the subsequent datafeed schedule. To change the datafeed schedule, please see the section titled "Altering the Datafeed Schedule".
From V3 of Intuitive for PaperCut, your only need to make scheduling changes in six places at the same time. These changes need to be made on lines 24, 41, 94, 138, 197 and 234, from 02:00 to your time of choice.
For example, if you'd like to push this forward by half an hour, you'll need to change the time to 02:30, as demonstrated below:
Similarly, should you wish to alter the days it runs, this can also be achieved by altering the contents of the "days" parameter. For example, to set it to run on Mondays, Wednesdays and Saturdays at 2:30, this can be written as such:
You should now have altered all six schedules in the ETL config. Save the ETL Config file and afterwards restart the ETL service.
Altering the Datafeed Schedule
To alter when the datafeeds load data from the database, each datafeed needs to be updated one at a time.
This can be changed through the Intuitive Dashboard UI by following the below process:
- Login to your dashboard
- Go to the menu icon in the top right , hover over Datafeed Builder then Edit Datafeed
- Select your target datafeed, i.e. Departmental Statistics - 12 Months
- Click the "Feed Scheduler" button
- On this screen, you can alter when the datafeeds will update. For example, this could pushed forward by half an hour, so we can change the time to 04:30 as shown below
- Save the changes, and repeat the process from step 2 for the remaining datafeeds.
