Build the Host Application and Xilinx FPGA Binary

Start your ec2 instance you have been using to develop your application and run the following command. This process will take about an hour. So, make sure your design is functionally correct (this can be done through sw/hw emulation) and you are satisfied with the performance (to reason about performance, please use hw emulation).

cd $VITIS_DIR/examples/xilinx/<YOUR_PROJECT_FOLDER> # Move to your project repo
make clean
make TARGET=hw DEVICE=$AWS_PLATFORM all

Set up AWS cli and S3 bucket

AWS CLI (You only need to do this step one time)

In your ec2 instance, run the following commands (ref):

curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip
sudo ./aws/install

Check if it’s properlly installed by running:

aws --version

Set your credentials, region and output. If you run the following command, it will ask you for Access Key ID and Secret Access Key which can be found in the email sent to you with the title “[EE109] AWS F1 resource”. For the region, write ‘us-east-1’ and for the output write ‘json’.

aws configure

The result should look like:

[centos@ip-172-31-21-2 src]$ aws configure
AWS Access Key ID [None]: <Your Access Key>
AWS Secret Access Key [None]: <Your Secret Access Key>
Default region name [None]: us-east-1
Default output format [None]: json

S3 bucket (You should go through this process whenever you want to make a new S3 bucket)

This S3 bucket will be used by the AWS scripts to upload your DCP to AWS for AFI generation which will be packaged into a tar file. Start by creating a bucket. Bucket names must be between 3 (min) and 63 (max) characters long. Bucket names can consist only of lowercase letters, numbers, dots (.), and hyphens (-). As a bucket name has to be unique, we recommend including your user name. For example, I used ginasohn.helloworld to run the hello world example.

aws s3 mb s3://<bucket-name> --region us-east-1  # Create an S3 bucket (choose a unique bucket name)
touch FILES_GO_HERE.txt                          # Create a temp file
aws s3 cp FILES_GO_HERE.txt s3://<bucket-name>/<dcp-folder-name>/  # Choose a dcp folder name 

The AFI creation process will generate logs and will be placed in your S3 bucket. These logs can be used for debug if the AFI generation fails. Next, create a folder for your log files:

touch LOGS_FILES_GO_HERE.txt                     # Create a temp file
aws s3 cp LOGS_FILES_GO_HERE.txt s3://<bucket-name>/<logs-folder-name>/  # Choose a logs folder name

Once your AFI has been created successfully, you are free to delete the tar file and logs as needed. Deleting these files will not delete or modify your AFI.

Create an Amazon FPGA Image (AFI)

At this point, we assume you have:

Validated your code using SW/HW Emulation and you are ready to create an AFI and test on F1.
Compiled your host application and Xilinx FPGA Binary
Setup AWS CLI and S3 bucket for AFI creation

The create_vitis_afi.sh script is provided to facilitate AFI creation from a Xilinx FPGA Binary, it:

Takes in your Xilinx FPGA Binary *.xclbin file
Calls aws ec2 create_fpga_image to generate an AFI under the hood
Generates a _afi_id.txt which contains the identifiers for your AFI
Creates an AWS FPGA Binary file with an *.awsxclbin extension that is composed of: Metadata and AGFI-ID.
- This *.awsxclbin is the AWS FPGA Binary file that will need to be loaded by your host application to the FPGA

To do so, run the following command (Here the output_aws_fpga_binary_awsxclbin_filename_root has to be the same name as your xclbin file):

cd $VITIS_DIR/examples/xilinx/<Your_Project_Folder>  # Moving to your project folder
$VITIS_DIR/tools/create_vitis_afi.sh -xclbin=<input_xilinx_fpga_binary_xclbin_filename>
    -o=<output_aws_fpga_binary_awsxclbin_filename_root> \
    -s3_bucket=ginasohn.helloworld -s3_dcp_key=dcp-folder -s3_logs_key=logs-folder

$VITIS_DIR/tools/create_vitis_afi.sh -xclbin=./build_dir.hw.xilinx_aws-vu9p-f1_shell-v04261818_201920_3/vadd.xclbin -o=vadd -s3_bucket=ginasohn.helloworld -s3_dcp_key=dcp-folder -s3_logs_key=logs-folder

Tracking the status of your registered AFI (ref)

Once you did the previous step successfully, the AFI creation has started. Let’s track the status of the AFI creation.

The *_afi_id.txt file generated by the create_vitis_afi.sh also includes the two identifiers for your AFI:

FPGA Image Identifier or AFI ID: this is the main ID used to manage your AFI through the AWS EC2 CLI commands and AWS SDK APIs. This ID is regional, i.e., if an AFI is copied across multiple regions, it will have a different unique AFI ID in each region. An example AFI ID is afi-06d0ffc989feeea2a.
Global FPGA Image Identifier or AGFI ID: this is a global ID that is used to refer to an AFI from within an F1 instance. For example, to load or clear an AFI from an FPGA slot, you use the AGFI ID. This is embedded into the AWS FPGA Binary *.awsxclbin file generated by create_vitis_afi.sh. Since the AGFI IDs is global (by design), it allows you to copy a combination of AFI/AMI to multiple regions, and they will work without requiring any extra setup. An example AGFI ID is agfi-0f0e045f919413242.

We will use the describe-fpga-images API and AFI ID to check the AFI state during the background AFI generation process.

aws ec2 describe-fpga-images --fpga-image-ids <AFI ID>

If the afi creation is not completed yet, you will see an output like:

[centos@ip-172-31-21-2 hello_world]$ aws ec2 describe-fpga-images --fpga-image-ids afi-075e81f7ead197bf9
{
    "FpgaImages": [
        {
            "FpgaImageId": "afi-075e81f7ead197bf9",
            "FpgaImageGlobalId": "agfi-07cbb8b1b71c293d9",
            "Name": "vadd",
            "Description": "vadd",
            "ShellVersion": "0x04261818",
            "PciId": {
                "DeviceId": "0xf010",
                "VendorId": "0x1d0f",
                "SubsystemId": "0x1d51",
                "SubsystemVendorId": "0xfedd"
            },
            "State": {
                "Code": "pending"
            },
            "CreateTime": "2024-05-20T06:49:19+00:00",
            "UpdateTime": "2024-05-20T07:39:05+00:00",
            "OwnerId": "891377270430",
            "Tags": [],
            "Public": false,
            "DataRetentionSupport": false,
            "InstanceTypes": [
                "f1.2xlarge",
                "f1.4xlarge",
                "f1.16xlarge"
            ]
        }
    ]
}

When AFI creation completes successfully, the output should contain:

    ...
    "State": {
        "Code": "available"
    },
    ...

The AFI creation usually takes several hours. After you have submitted your bitstream for AFI creation, you don’t have to keep your instance on. You can check if the AFI creation is complete by running the following command in your local machine (To do so, write down your <AFI ID>). However, to do so, you will first have to:

Install AWS CLI in your local machine. Follow the instructions in [this link] (https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html).
Set your credentials, region and output. If you run the following command, it will ask you for Access Key ID and Secret Access Key which can be found in the email sent to you with the title “FPGA Access Key”. For the region, write ‘us-east-1’ and for the output write ‘json’.
```
 aws configure
```
The result should look like:
```
 [centos@ip-172-31-21-2 src]$ aws configure
 AWS Access Key ID [None]: <Your Access Key>
 AWS Secret Access Key [None]: <Your Secret Access Key>
 Default region name [None]: us-east-1
 Default output format [None]: json
```

Run the following command:

 aws ec2 describe-fpga-images --fpga-image-ids <AFI ID>

If the “State” code indicates the AFI generation has “failed”, the AFI creation logs can be found in the bucket location (s3://<bucket-name>/<logs-folder-name>) provided to create_vitis_afi.sh above. These will detail the errors encountered during the AFI creation process.

For help with AFI creation issues, see create-fpga-image error codes.

Run the FPGA accelerated application on Amazon FPGA instances

Start your AWS F1 instance (Note this is different from the instance you have been using to do the steps above). The designated instances for your team and their ssh key can be found in the email with the title “[EE109] AWS F1 resource”.

If it’s the first time you are starting the instance, clone the github repository to the new F1 instance:

git clone https://github.com/aws/aws-fpga.git $AWS_FPGA_REPO_DIR

Install runtime drivers:

cd $AWS_FPGA_REPO_DIR
source vitis_runtime_setup.sh  # You need to run the following steps everytime you start the instance or a new terminal

Run the following command and see if the MPD service has been initialized.

systemctl status mpd

If you see the following result, you are good to proceed to the next step:

[centos@ip-172-31-84-161 aws-fpga]$ systemctl status mpd
● mpd.service - Xilinx Management Proxy Daemon (MPD)
   Loaded: loaded (/etc/systemd/system/mpd.service; disabled; vendor preset: disabled)
   Active: active (running) since Mon 2024-05-20 14:18:20 UTC; 6s ago
 Main PID: 2439 (mpd)
   CGroup: /system.slice/mpd.service
           └─2439 /opt/xilinx/xrt/bin/mpd

Assuming the developer flow (compilation) was done on a separate build instance, you will need to:

Copy the compiled host executable (exe) to the new F1 instance
- This will have the same name as the project folder name (e.g., for the hello world example this is the “hello_world.exe” file)
Copy the *.awsxclbin AWS FPGA binary file to the new instance
Copy any data files required for execution to the new instance

You can copy these files under $AWS_FPGA_REPO_DIR.

To execute your Host Application:

chmod +x <compiled-host-executable>
./<compiled-host-executable> ./<awsxclbin-file-of-your-app>

If the FPGA is properlly programmed and your application runs successfully, you will see an output message in your terminal similar to this:

[centos@ip-172-31-84-161 aws-fpga]$ chmod +x hello_world 
[centos@ip-172-31-84-161 aws-fpga]$ ./hello_world ./vadd.awsxclbin
Found Platform
Platform Name: Xilinx
INFO: Reading ./vadd.awsxclbin
Loading: './vadd.awsxclbin'
Trying to program device[0]: xilinx_aws-vu9p-f1_dynamic-shell
Device[0]: program successful!
TEST PASSED