File Server Connector

The File Server connector is used to process files in the specified source directory. Note that files cannot remain in the source directory after processing or they will be processed again. Therefore, after processing a file it will be deleted.

Defining a file service

Step 1: Create the service

Create a service with unique name.

service orderProcessService {

}

Step 2: Specify service parameters

Add a service level annotation named file:FileSource and add the key-value pairs to specify the parameters. The following section describes each key that can be used with a file service. An example is provided after the tables.

Key Description Required Expected Value Default value
protocol The protocol to which this service is bound. Yes file -
fileURI The URI where the files you want to process are located. This can be a URI to a folder. If it is a folder, all the files in the folder will be processed, one at a time. If the URI points to a single file, the file will be processed when it becomes available at that location. Yes A valid file URI.

If the file URI contains a user name or a password, make sure that it does not contain '#' or '?' character.
-
pollingInterval The polling interval at which the file URI location is polled for the file. The value is expressed in milliseconds. No A positive integer. 10000
acknowledgementTimeOut The expected time in milliseconds that a file needs to stay open until the consumer has finished consuming it. When you develop the Ballerina service, if you are performing time-consuming operations that require the file to be kept open (for example, streaming the file content to a slow backend), this value needs to be set to a higher value accordingly. No A positive integer. 30000
deleteIfNotAcknowledged If this parameter is set to "true", the file will be deleted in case the file consumer did not acknowledge. Otherwise (if set to "false") the file will be kept without deleting, so it will be retried to be processed again in the next polling cycle. No true or false false

When the fileURI parameter points to a folder, the user has the option to sort the files, which are currently in this folder, before processing starts. Following are the parameters you can use to configure sorting operations.

Key Description Required Expected Value Default value
fileSortAttribute The attribute of the file by which the files will be sorted. No One of the following: name, size, lastModifiedTimestamp -
fileSortAscending A Boolean parameter that indicates whether to sort files in ascending order. If set to "true", files will be sorted in ascending order. If set to “false”, files will be sorted in descending order. No true or false true

Example:

@file:FileSource (
  fileURI : "file:///home/user/orders",
  pollingInterval : "20000",
  fileSortAttribute : "size",
  fileSortAscending : "false"
  )
service orderProcessService {
}

Step 3: Add a resource

Add a resource under the file service as below:

@file:FileSource (
    fileURI : "file:///home/user/orders",
    pollingInterval : "20000",
    fileSortAttribute : "size",
    fileSortAscending : "false"
    )
service orderProcessService { 
    resource processOrder (message m) {
        // file processing logic here.
    }
}

In general, a service can have multiple resources. However, a service of type File is required to have one and only one resource.

Step 4: Add file-processing logic

Within the resource block, specify the file-processing logic. In the example given below, Ballerina functions (system:println, messages:getStringPayload(m), and file:acknowledge(m)) are being used to process the file.

import ballerina.lang.messages;
import ballerina.lang.system;
import ballerina.net.file;

@file:FileSource (
  fileURI : "file:///home/user/orders",
  pollingInterval : "20000",
  fileSortAttribute : "size",
  fileSortAscending : "false"
  )
service orderProcessService {
    resource processOrder (message m) {
        system:println(messages:getStringPayload(m));
        file:acknowledge(m);
    }
}

Note: Here, file:acknowledge(m) is a function that is exclusive for file processing. See the function description below for details.

Step 5: Add dependency JARs

When the fileURI parameter refers to a location in the local file system, you do not need to add any additional JARs for the file service to work. However, in other cases (for example, when the fileURI refers to a remote file or a folder that needs to be accessed via FTP), it may be required to add specific JARs to the <ballerina_home>/bre/lib folder.

The following table indicates which dependency JARs are required for which file-access protocol.

Dependency Required For
Commons Compress Version 1.9. TAR, Bzip2
Commons Net Version 3.3. FTP
Commons Httpclient Version 3.1. Requires Commons Codec Version 1.2. HTTP, URI Utils
JSch Version 0.1.51. SFTP

Native Ballerina functions for file processing

Acknowledge

Function name: acknowledge

Package name: ballerina.net.file

Arguments: message

The acknowledge function sends an acknowledgement to the sender of the message, saying that file processing has being finished.

It is important to note that this function must be called after the service has finished consuming the message (that is, the file). See below for an example.

import ballerina.lang.messages;
import ballerina.lang.system;
import ballerina.net.file;

@file:FileSource (
  fileURI : "file:///home/user/orders",
  pollingInterval : "20000",
  fileSortAttribute : "size",
  fileSortAscending : "false"
  )
service orderProcessService {
    resource processOrder (message m) {
        system:println(messages:getStringPayload(m));
        file:acknowledge(m);
    }
}

In the above example, once a file is found at the given URI, orderProcessService will receive a message. This message contains a reference to the input stream of the file so the service can process the file.

The service will then execute the statements given within the resource block.

system:println(messages:getStringPayload(m));

The above line will read the file content as a String and then print it on the console.

file:acknowledge(m);

The above line sends an acknowledgment to the sender of the message (this sender has the control to close the file input stream). As a result, the file input stream will be closed, and then the file will be deleted.

Since this function makes the message sender close the input stream and delete the file, this function needs to be called only after message processing is done.

In case the service does not call the acknowledge function, the message sender will wait for 30 seconds (30 seconds is the default wait time. This value can be overridden by specifying a different value as the acknowledgementTimeOut service parameter) and assume that the file was not processed. Following the same assumption, the message sender will not delete the file. As a result of this, the file will remain at the same URI to which the service listens, so it will be attempted to be processed in the next polling cycle as well. This behavior can be changed by setting deleteIfNotAcknowledged service parameter to "true". If it is set to "true" then, the file will be deleted anyway, regardless of whether the acknowledgement was made or not.