Skip to content

Federated Learning

ARCC Documentation

Federated Learning is one of the newest approaches to machine learning leveraging techniques for decentralization in which a global model is trained across multiple devices and servers (clients) without needing to ever exchange raw data.

As opposed to traditional machine learning training in which the full dataset must be moved to and available to a central server, the model is sent to localized data. This ensures that sensitive information remains on the local device or within the confines of local secure environments.

Advantages

As described above, Federated Learning offers a number of advantages. We list several below:

  1. It provides a method for developing and validating more robust, accurate, and generalizable AI models from diverse data sources.

  2. By sharing the model across data sources instead of sharing the data, risks associated with data security and/or privacy compromises is mitigated. This is especially valuable in cases where we are using restricted data in the training of a model.

    • This removes the data silos associated with traditional machine learning by centrally aggregating the intermediate results from the trained model instead of the raw underlying data, itself.

    • This removes the hurdles associated with creating more robust and powerful machine learning models, since many types of data are subject to restrictions that impeded training, like privacy laws, size of data sets, and other regulations or restrictions associated with access and use of data.

    • Compliance is maintained in situations subject to data sovereignty and/or industry regulation.

  3. Sharing the model allows shared knowledge in the form of trained model updates made between federated endpoints which are often smaller in size than the larger datasets that would need to be shared to come up with the models.

  4. Federated learning utilizes multiple security features at different layers, including homomorphic encryption, differential privacy, and confidential computing.

NVIDIA Flare

NVIDIA Flare is NVIDIA's Proprietary Federated Learning Toolkit. FLARE stands for Federated Learning Application Runtime Environment. This is a "domain-agnostic, open-source, extensible SDK that allows researchers and data scientists to adapt existing ML/DL workdlows (PyTorch, RAPIDS, Nemo, TensorFlow) to a federated paradigm; and enables platofrm developers to build a secure, privacy preserving offering for a distributed multi-party collaboration."

Federated Learning

Workflow for Running NVFlare on an ARCC HPC System

Assumptions

The following provides direction for how to configure and use NV FLare on ARCC HPC Systems with some caveats/assumptions in our particular example:

  1. The core federated learning software stack is PyTorch

  2. We assume the federated learning administrative client is running from an ARCC managed VM, named [nv-fl-aggreg].

    1. For this specific example you will need to manage your own administration client or request an ARCC VM from which you can manage your federated learning.

    2. This system must have prerequisites installed including miniconda3, NV Flare, and several other Python related packages.

  3. We assume one of the several connected clients in which the model is distributed includes UW ARCC's Medicinebow cluster.

    1. In order for your FL Administrative Client to send jobs to the Medicinebow cluster, you must have ARCC cluster access to be able to authenticate to Medicinebow and run your jobs.

Example: The General Steps

  • 1. Set up your Primary Server

    Beginning on the primary server:

    1. Access the server
    2. Set it up
    3. Clone the NVFlare Repo

    Go to step 1

  • 2. Provision a Startup Kit

    From the server:

    1. Generate the default template
    2. Configure the template
    3. Generate the startup kit
    4. Configure startup kit files
    5. Package kit for client deplotment

    Go to step 2

  • 3. Set up on Each Client(s)

    From server, for each client:

    1. Retrieve the kit from your server
    2. Install/Configure Prerequisites
    3. Clone NV Flare Repo
    4. Install NV Flare Requirements
    5. Set up dataset

    Go to step 3

  • 4. Create a Job from Admin

    To be done on each client:

    1. Configure job template(s)
    2. Create a job
    3. Configure for server and client(s)
    4. Define model scripts for server and client(s)

    Go to Step 4

  • 5. Submit and Run Job(s) using Admin

    From the primary server:

    1. Start the service from Admin Server
    2. Start service and job(s) on each client
    3. Confirm connections
    4. Start Admin service
    5. Confirm all servers, clients, and associated services & jobs are running
    6. Submit jobs using Admin Client

    Go to Step 5

  • 6. Shut Down Client(s) and Server(s)

    After confirming jobs are completed and appropriate work is saved:

    1. Shut down each client
    2. Shut down server
    3. Check for lingering processes
    4. Any remaining cleanup

    Finish with Step 6

1. Set up your Primary Server/VM

Before starting, you'll need to have an ARCC account from which you're able to run jobs, and a central client upon which you can run your management server. In our example, this management server is run from an ARCC managed VM, nv-fl-aggreg. If planning to set up your management system with ARCC, you will need to request a VM environment with access using your ARCC/UW account.

  1. Accessing the Flare Server (Assuming this is an ARCC hosted system)

    1. Access using your ssh keys with the following command:

      [~]$ ssh -i ~/.ssh/nvflare/id-ecdsa username@nv-fl-aggreg.arcc.uwyo.edu
username@nv-fl-aggreg:~$

      If running from an ARCC hosted VM, make note of Operating System

      The system will be running the most recent version of Ubuntu LTS unless otherwise specified.

    2. Install Miniconda3:
      Since we do not have Miniconda installed by default on the system, we will need to install it.

      Expand for commands to install miniconda if necessary 
      username@nv-fl-aggreg:~$ wget https:// ://repo.anaconda.com/miniconda/Miniconda3-py312_25.9.1-3-Linux-x86_64.sh 
username@nv-fl-aggreg:~$ sh Miniconda3-py312_25.9.1-3-Linux-x86_64.sh 
Welcome to Miniconda3 py312_25.9.1-3 
In order to continue the installation process, please review the license
agreement. 
Please, press ENTER to continue 
>>> 
MINICONDA END USER LICENSE AGREEMENT 
... 
Do you accept the license terms? [yes|no] 
>>> yes 
Miniconda3 will now be installed into this location: 
/home/username/miniconda3 
... 
installation finished. 
Do you wish to update your shell profile to automatically initialize conda? 
... 
Proceed with initialization? [yes|no] 
[no] >>> no 
You have chosen to not have conda modify your shell scripts at all. 
To activate conda's base environment in your current shell session: 
eval "$(/home/username/miniconda3/bin/conda shell.YOUR_SHELL_NAME hook)" 
To install conda's shell functions for easier access, first activate, then: 
conda init 
Thank you for installing Miniconda3!  
username@nv-fl-aggreg:~$

    3. Once it's installed, update your .bashrc script to add miniconda commands to the PATH and configure related environment variables: 

      username@nv-fl-aggreg:~$ vim .bashrc 
export PATH=/home/username/miniconda3/bin:$PATH 
source /home/username/miniconda3/etc/profile.d/conda.sh 

# Check conda is on PATH 
userhelp@nv-fl-aggreg:~$ exec bash 
userhelp@nv-fl-aggreg:~$ conda --version 
conda 25.9.1

    4. Clone the NV Flare Repository: 

      username@nv-fl-aggreg:~$ git clone https://github.com/NVIDIA/NVFlare.git 
Cloning into 'NVFlare'... 

# The repository will look something like:  
username@nv-fl-aggreg:~$ ls NVFlare/ 
3rdParty         ci                  CODEOWNERS          CONTRIBUTING.md  examples       LICENSE       nvflare_nightly.egg-info  README.md         runtest.sh  tests
build_docker.sh  CITATION.cff        commit_message.txt  docker           integration    MANIFEST.in   nvflight                  requirements.txt  setup.cfg   versioneer.py
build_doc.sh     CODE_OF_CONDUCT.md  conftest.py         docs             job_templates  nvflare        pyproject.toml            research          setup.py    web

    5. Create the Conda environment, then install NV Flare and other required Python packages

      Be aware that NV Flare requires a Python version between 3.8 and 3.12

      As of this time, no other versions are supported and unsupported versions will result in runtime error.

      Expand for commands to install miniconda if necessary 
      username@nv-fl-aggreg:~$ conda create -p nvflare_env python=3.12
# The first time you using conda, the following messags will be presented.
Do you accept the Terms of Service (ToS) for https://repo.anaconda.com/pkgs/main? [(a)ccept/(r)eject/(v)iew]: a
Do you accept the Terms of Service (ToS) for https://repo.anaconda.com/pkgs/r? [(a)ccept/(r)eject/(v)iew]: a
2 channel Terms of Service accepted
...
# To activate this environment, use
#     $ conda activate /home/username/nvflare_env
# To deactivate an active environment, use
#     $ conda deactivate

username@nv-fl-aggreg:~$ conda activate /home/username/nvflare_env
# Forces all python packages to be installed within the conda environment location.
(/home/username/nvflare_env) username@nv-fl-aggreg:~$ export PYTHONUSERBASE=$CONDA_PREFIX
(/home/username/nvflare_env) username@nv-fl-aggreg:~$ cd NVFlare/
(/home/username/nvflare_env) username@nv-fl-aggreg:~/NVFlare$ pip install -e ".[PT,SKLEARN]"
(/home/username/nvflare_env) username@nv-fl-aggreg:~/NVFlare$ nvflare --version
NVFlare version is 2.7.0+22.g02ae6e17
(/home/username/nvflare_env) username@nv-fl-aggreg:~/NVFlare$ conda deactivate
username@nv-fl-aggreg:~/NVFlare$

    6. For convenience of using the Flare environment, and for it to activate at login: 

      username@nv-fl-aggreg:~$ vim .bashrc
conda activate /home/username/nvflare_env
export PYTHONUSERBASE=$CONDA_PREFIX
export PYTHONPATH=/home/username/NVFlare

    7. Confirm command line tools by starting a new session logging into the system and check that the environment activates and NV Flare commands are available.

      (/home/username/nv_flare_env) username@nv-fl-aggreg:~$ 
(/home/username/nvflare_env) username@nv-fl-aggreg:~$ nvflare --version
NV Flare version is 2.7.0+22.g02ae6e17

2. Provision a Startup Kit for a Project

We will be setting this up with the command line (CLI) provision command

As an alternative, you can also use the dashboard UI, but that will not be covered in this example/walkthrough.

  1. Create our Project Template: 

    ~$ mkdir demo_env
~$ cd demo_env/
~/demo_env$ nvflare provision -g
/data/home/username/demo_env/project.yml was generated.  Please edit it to fit your NVFlare configuration. 
Once done please run 'nvflare provision -p project.yml' to perform the provisioning
(/home/username/nvflare_env) username@nv-fl-aggreg:~/demo_env$ ls
project.yml
    This will generate a template called project.yml

  2. Update our project.yml file Before we can provision our project startup kit, we must update default values within the yml file:

    Expand to review the updated .yml file 
    api_version: 3
name: demo01
description: NVIDIA FLARE sample project yaml file

participants:
# Change the name of the server (server1) to the Fully Qualified Domain Name
# (FQDN) of the server, for example: server1.example.com.
# Ensure that the FQDN is correctly mapped in the /etc/hosts file.
- name: fl-junction.arcc.uwyo.edu
    type: server
    org: arcc
    fed_learn_port: 8002
    admin_port: 8003
- name: medbow1
    type: client
    org: arcc
- name: medbow2
    type: client
    org: arcc
- name: admin@uwyo.edu
    type: admin
    org: arcc
    role: project_admin

# The same methods in all builders are called in their order defined in builders section
builders:
- path: nvflare.lighter.impl.workspace.WorkspaceBuilder
    args:
    template_file:
        - master_template.yml
- path: nvflare.lighter.impl.static_file.StaticFileBuilder
    args:
    config_folder: config
- path: nvflare.lighter.impl.cert.CertBuilder
- path: nvflare.lighter.impl.signature.SignatureBuilder

  3. Provision and generate the project startup kit 

    (/home/<username>/nvflare_env) <username>@fl-junction:~/demo_env$ nvflare provision -p project.yml
Project yaml file: /data/home/<username>/demo_env/project.yml.
INFO: Generated results can be found under /data/home/<username>/demo_env/workspace/demo01/prod_00.

# Creates the workshop folder that contains keys and scripts for the server, admin cleint and a folder for evry client.
(/home/<username>/nvflare_env) <username>@fl-junction:~/demo_env$ ls
project.yml  workspace

    Notice

    1. There are separate folders for the server, admin client, and one for each defined client.
    2. Each has a startup folder contianing the start up and shutdown related scripts.
    3. Security related keys and configurations are automatically generated and shouldn't be manually changed.
    4. Clients can have resource.json configuration files defined.
    Expand to show example dirs and files generated Under Workspace: 
    # List of some of the key files and folders generated.
  workspace
  ├── demo01/
  │   ├── prod00/
  │   │   └── admin@uwyo.edu/                  # Matches the name defined within project.yml
  │   │       └── local/
  │   │       └── startup/
  │   │           └── client.crt
  │   │           └── client.key
  │   │           └── fed_admin.json
  │   │           └── fl_admin.sh              # Script to call to start the admin client
  │   │           └── readme.txt
  │   │           └── rootCA.pem
  │   │           └── signature.json
  │   │           └── system_info.ipynb
  │   │       └── transfer
  │   │   └── fl-junction.arcc.uwyo.edu/       # Matches the FQDN name defined within project.yml
  │   │       └── local/
  │   │       └── readme.txt
  │   │       └── startup/
  │   │           └── fed_server.json
  │   │           └── rootCA.pem
  │   │           └── server.crt
  │   │           └── server.key
  │   │           └── signature.json
  │   │           └── start.sh                 # Script to call to start the FL server.
  │   │           └── stop_fl.sh               # Script to call to shutdown the FL server.
  │   │           └── sub_start.sh
  │   │       └── transfer
  │   │   └── medbow01/
  │   │       └── local/
  │   │           └── authorization.json.default  
  │   │           └── log_config.json.default  
  │   │           └── privacy.json.sample  
  │   │           └── resources.json.default   # File used to define GPU resources if available.
  │   │           └── signature.json
  │   │       └── readme.txt
  │   │       └── startup/
  │   │           └── client.crt
  │   │           └── client.key
  │   │           └── fed_client.json
  │   │           └── rootCA.pem
  │   │           └── signature.json
  │   │           └── start.sh                 # Script to call to start the first client.
  │   │           └── stop_fl.sh               # Script to call to shutdown the first client.
  │   │           └── sub_start.sh
  │   │       └── transfer
  │   │   └── medbow02/
  │   │       └── local/
  │   │       └── readme.txt
  │   │       └── startup/
  │   │           └── start.sh                 # Script to call to start the second client.
  │   │           └── stop_fl.sh               # Script to call to shutdown second first client
  │   │       └── transfer
  │   └── state/
  │       └── cert.json

  4. Package the Project Startup Kit

    Since the project related folders have details for the entire environment we need to package it up so it can be copied over to the clients later.

    Expand to review the commands for packaging our environment: 
    (/home/username/nvflare_env) username@nv-fl-aggreg:~ tar -cf demo01.tar demo_env/
(/home/username/nvflare_env) username@nv-fl-aggreg:~$ ls
demo01.tar  
demo_env
miniconda3
Miniconda3-py312_25.9.1-3-Linux-x86_64.sh
.nvflare
NVFlare   
nvflare_env

3. Set up the Environment on Each Client

PyTorch Example

Note we will be using an environment and running a job based on a Hello PyTorch example.
Some of the details and files are listed on the corresponding web page but may not be available within the repository.

  1. Retrieve the Project Startup Kit from the "Server/VM" By design, we can only retrieve files from the original aggregator source:

    # On MedicineBow, navigate to an existing project's software folder.
# For this example we are using /project/arcc-nv-flare/software/
[software]$ scp -i ~/.ssh/nvflare/id_ecdsa username@nv-fl-aggreg.arcc.uwyo.edu:/home/username/demo01.tar .
[software]$ tar -xf demo01.tar
[software]$ ls demo_env/
project.yml  workspace
[username@mblog2 software]$ ls demo_env/workspace/demo01/prod_00/
admin@uwyo.edu  nv-fl-aggreg.arcc.uwyo.edu  medbow1  medbow2

  2. Install Miniconda3 if not already available On Medicinebow, Miniconda3 is already installed as a module, so we can simply load it:
    module load miniconda3

  3. Clone NV Flare repository and create the nvflare Conda environment

    We can do this using the same process as we used on the original server. Expand for example 
    # Under: /project/arcc-nv-flare/software/
[]$ git clone https://github.com/NVIDIA/NVFlare.git
[]$ conda create -p demo_client_env python=3.12
[]$ conda activate /cluster/medbow/project/arcc-nv-flare/software/demo_client_env
()[]$ export PYTHONUSERBASE=$CONDA_PREFIX
()[]$ cd NVFlare/
()[]$ pip install -e ".[PT,SKLEARN]"
()[]$ nvflare --version
NVFlare version is 2.7.0+22.g02ae6e17
    1. Install Client Related Requirements for NV Flare 
      # Now install the client specific dependencies.
()[]$ cd examples/hello-world/hello-pt/
()[]$ pip install -r requirements.txt
# This is an example where the web page relating to the example is different.
# The 'safetensors' package is required when running the code, but there is no reference to it anywhere.
()[]$ pip install safetensors
()[]$ nvflare --version
NVFlare version is 2.7.1
    2. Set up our Data This example uses the CIFAR-10 dataset. To use it, we'll first copy it over: 
      # Under: /project/arcc-nv-flare/software/
()[]$ mkdir data
()[]$ cd data
()[]$ cp /project/arcc-nv-flare/software/NVFlare/examples/hello-world/step-by-step/cifar10/data/download.py .
()[]$ vim download.py
# CIFAR10_ROOT = "/tmp/nvflare/data/cifar10"
CIFAR10_ROOT = "./cifar10"
()[]$ python download.py
100.0%
()[]$ ls
cifar10  download.py

    Note the location of the data with respect to each client

    The data location may be different from system to system.

4. Create a Job using the Admin Client

To do this, we'll use the Job CLI.

  1. Configure the Job Templates To use the job related CLI, we must first configure where the nv-flare job templates can be found within the cloned repo.

    # Under your home folder (~), if it doesn't already exist, create the .nvflare folder.
()$ cat .nvflare/config.conf
{
    job_template {
    path = "/home/<username>/NVFlare/job_templates"
    }
}

  2. Create the job Flare offers a number of job templates dependent upon the workflow being used. The command below may be expanded to show an example list of available job templates.

    Expand to review a list of available job templates 
    ()$ nvflare job list_templates
The following job templates are available:
------------------------------------------------------------------------------------------------------------------------
    name                 Description                                                  Controller Type   Execution API Type
------------------------------------------------------------------------------------------------------------------------
    cyclic_cc_pt         client-controlled cyclic workflow with PyTorch ClientAPI tra client            client_api
    cyclic_pt            server-controlled cyclic workflow with PyTorch ClientAPI tra server            client_api
    fedopt_pt            FedOpt using pytorch with Client API                         server            client_api
    psi_csv              private-set intersection for csv data                        server            Executor
    sag_cross_np         FedAvg and cross-site validation using numpy                 server            client executor
    sag_cse_ccwf_pt      Client Controller FedAvg and cross-site evaluation with PyTo client            client_api
    sag_cse_pt           FedAvg and cross-site evaluation with PyTorch                server            client_api
    sag_gnn              FedAvg with gnn learning                                     server            client_api
    sag_nemo             FedAvg for NeMo                                              server            client_api
    sag_np               FedAvg with numpy                                            server            client_api
    sag_np_cell_pipe     FedAvg with numpy                                            server            client_api
    sag_np_metrics       FedAvg with numpy and metrics relay                          server            client_api
    sag_pt               FedAvg using pytorch                                         server            client_api
    sag_pt_deploy_map    FedAvg with pytorch, deploy_map, site-specific configs       server            client_api
    sag_pt_executor      FedAvg, cross-site evaluation with PyTorch Executor          server            Executor
    sag_pt_he            FedAvg using pytorch and homomorphic encryption              server            client_api
    sag_pt_in_proc       FedAvg using pytorch with in_process Client API              server            client_api
    sag_pt_mlflow        FedAvg using pytorch with MLflow tracking                    server            client_api
    sag_tf               FedAvg using TensorFlow                                      server            client_api
    sklearn_kmeans       scikit-learn KMeans model                                    server            client_api
    sklearn_linear       scikit-learn linear model                                    server            client_api
    sklearn_svm          scikit-learn SVM model                                       server            client_api
    stats_df             FedStats: tabular data with pandas                           server            stats executor
    stats_image          FedStats: image intensity histogram                          server            stats executor
    swarm_cse_pt         Swarm Learning with Cross-Site Evaluation with PyTorch       client            client_api
    swarm_cse_pt_model_l Swarm Learning with Cross-Site Evaluation with PyTorch Model client            ModelLearner
    vertical_xgb         vertical federated xgboost                                   server            Executor
    xgboost              NA                                                           NA                NA
    xgboost_tree         xgboost horizontal tree-based collaboration model            server            client_api
------------------------------------------------------------------------------------------------------------------------
    Expand the example: Creating a template for the Scatter and Gather workflow 
    # Within your home folder:
()$ nvflare job create -j demo_job -w sag_pt
The following are the variables you can change in the template
---------------------------------------------------------------------------------------------------------------------------------------
    job folder: demo_job
---------------------------------------------------------------------------------------------------------------------------------------
    file_name                      var_name                       value                               component
---------------------------------------------------------------------------------------------------------------------------------------
    meta.conf                      app                            ['@ALL']
    meta.conf                      mandatory_clients              []
    meta.conf                      min_clients                    2
    config_fed_client.conf         app_config
    config_fed_client.conf         app_script                     cifar10.py
    config_fed_client.conf         component_ids                  ['metric_relay']                    ExternalConfigurator
    config_fed_client.conf         config_file_name               client_api_config.json              ExternalConfigurator
    config_fed_client.conf         evaluate_task_name             validate
    config_fed_client.conf         event_type                     fed.analytix_log_stats              MetricRelay
    config_fed_client.conf         external_pre_init_timeout      300.0
    config_fed_client.conf         fed_event                      True                                MetricRelay
    config_fed_client.conf         heartbeat_interval             5.0                                 MetricRelay
    config_fed_client.conf         heartbeat_timeout              60.0                                MetricRelay
    config_fed_client.conf         last_result_transfer_timeout   300.0
    config_fed_client.conf         launch_once                    True                                SubprocessLauncher
    config_fed_client.conf         mode                           PASSIVE                             CellPipe
    config_fed_client.conf         monitor_interval               0.01
    config_fed_client.conf         params_exchange_format         pytorch
    config_fed_client.conf         params_transfer_type           DIFF
    config_fed_client.conf         peer_read_timeout              300.0
    config_fed_client.conf         pipe_channel_name              metric                              MetricRelay
    config_fed_client.conf         read_interval                  0.5
    config_fed_client.conf         root_url                       {ROOT_URL}                          CellPipe
    config_fed_client.conf         script                         python3 -u custom/{app_script}      ap SubprocessLauncher
    config_fed_client.conf         secure_mode                    {SECURE_MODE}                       CellPipe
    config_fed_client.conf         server_expected_format         ExchangeFormat.NUMPY
    config_fed_client.conf         shutdown_timeout               0.0                                 SubprocessLauncher
    config_fed_client.conf         site_name                      {SITE_NAME}                         CellPipe
    config_fed_client.conf         token                          {JOB_ID}                            CellPipe
    config_fed_client.conf         train_with_evaluation          True
    config_fed_client.conf         workers                        4
    config_fed_client.conf         workspace_dir                  {WORKSPACE}                         CellPipe
    config_fed_server.conf         allow_empty_global_weights     False                               ScatterAndGather
    config_fed_server.conf         allow_numpy_conversion         True                                PTFileModelPersistor
    config_fed_server.conf         best_global_model_file_name    best_FL_global_model.pt             PTFileModelPersistor
    config_fed_server.conf         events                         ['fed.analytix_log_stats']          TBAnalyticsReceiver
    config_fed_server.conf         expected_data_kind             WEIGHT_DIFF                         InTimeAccumulateWeightedAggregator
    config_fed_server.conf         global_model_file_name         FL_global_model.pt                  PTFileModelPersistor
    config_fed_server.conf         ignore_result_error            False                               ScatterAndGather
    config_fed_server.conf         key_metric                     accuracy                            IntimeModelSelector
    config_fed_server.conf         load_weights_only              False                               PTFileModelPersistor
    config_fed_server.conf         min_clients                    2                                   ScatterAndGather
    config_fed_server.conf         model_class_path               net.Net
    config_fed_server.conf         negate_key_metric              False                               IntimeModelSelector
    config_fed_server.conf         num_rounds                     2                                   ScatterAndGather
    config_fed_server.conf         persist_every_n_rounds         1                                   ScatterAndGather
    config_fed_server.conf         snapshot_every_n_rounds        1                                   ScatterAndGather
    config_fed_server.conf         start_round                    0                                   ScatterAndGather
    config_fed_server.conf         task_check_period              0.5                                 ScatterAndGather
    config_fed_server.conf         tb_folder                      tb_events                           TBAnalyticsReceiver
    config_fed_server.conf         train_timeout                  0                                   ScatterAndGather
    config_fed_server.conf         validation_metric_name         initial_metrics                     IntimeModelSelector
    config_fed_server.conf         wait_time_after_min_received   0                                   ScatterAndGather
    config_fed_server.conf         weigh_by_local_iter            False                               IntimeModelSelector
---------------------------------------------------------------------------------------------------------------------------------------
    Expand this example to see job directory structure 
        demo_job
├── app/
│   ├── config/
│   │   └── config_fed_client.conf
│   │   └── config_fed_server.conf
│   ├── custom/
│   │   └──       # Under this folder you will create the client and model scripts.
├── meta.conf     # Specifies the minimum number of clients required by the job.
()$ cd ~/demo_job/
# Confirm that 'min_clients' is set to 2
()$ cat meta.conf
    min_clients = 2
    Replace the generated *.conf files: config_fed_client.conf 
    {
    "format_version": 2,
    "executors": [
        {
            "tasks": [
                "*"
            ],
            "executor": {
                "path": "nvflare.app_opt.pt.in_process_client_api_executor.PTInProcessClientAPIExecutor",
                "args": {
                    "task_script_path": "client.py"
                }
            }
        }
    ],
    "components": [
        {
            "id": "event_to_fed",
            "path": "nvflare.app_common.widgets.convert_to_fed_event.ConvertToFedEvent",
            "args": {
                "events_to_convert": [
                    "analytix_log_stats"
                ]
            }
        }
    ],
    "task_data_filters": [],
    "task_result_filters": []
}
    Replace the generated *.conf files: config_fed_server.conf 
    {
    "format_version": 2,
    "workflows": [
        {
            "id": "controller",
            "path": "nvflare.app_common.workflows.scatter_and_gather.ScatterAndGather",
            "args": {
                "min_clients": 2,
                "num_rounds": 2,
                "wait_time_after_min_received": 0,
                "persistor_id": "persistor"
            }
        }
    ],
    "components": [
        {
            "id": "json_generator",
            "path": "nvflare.app_common.widgets.validation_json_generator.ValidationJsonGenerator",
            "args": {}
        },
        {
            "id": "model_selector",
            "path": "nvflare.app_common.widgets.intime_model_selector.IntimeModelSelector",
            "args": {
                "aggregation_weights": {},
                "key_metric": "accuracy"
            }
        },
        {
            "id": "receiver",
            "path": "nvflare.app_opt.tracking.tb.tb_receiver.TBAnalyticsReceiver",
            "args": {
                "events": [
                    "analytix_log_stats",
                    "fed.analytix_log_stats"
                ]
            }
        },
        {
            "id": "persistor",
            "path": "nvflare.app_opt.pt.file_model_persistor.PTFileModelPersistor",
            "args": {
                "model": {
                    "path": "model.SimpleNetwork",
                    "args": {}
                }
            }
        },
        {
            "id": "locator",
            "path": "nvflare.app_opt.pt.file_model_locator.PTFileModelLocator",
            "args": {
                "pt_persistor_id": "persistor"
            }
        },
        {
            "id": "shareable_generator",
            "path": "nvflare.app_common.shareablegenerators.full_model_shareable_generator.FullModelShareableGenerator",
            "args": {}
        },
        {
            "id": "aggregator",
            "path": "nvflare.app_common.aggregators.intime_accumulate_model_aggregator.InTimeAccumulateWeightedAggregator",
            "args": {
                "expected_data_kind": "WEIGHTS"
            }
        },
        {
            "id": "receiver1",
            "path": "nvflare.app_opt.tracking.tb.tb_receiver.TBAnalyticsReceiver",
            "args": {
                "events": [
                    "analytix_log_stats",
                    "fed.analytix_log_stats"
                ]
            }
        }
    ],
    "task_data_filters": [],
    "task_result_filters": []
}
    Define our client and model scripts 
    ()$ cd ../custom
()$ cp ~/NVFlare/examples/hello-world/hello-pt/client.py .
()$ cp ~/NVFlare/examples/hello-world/hello-pt/model.py .
()$ vim client.py
# Update: DATASET_PATH = "/tmp/nvflare/data"
# To    : DATASET_PATH = "/project/arcc-nv-flare/software/data/cifar10"

5. Submit and Run Job(s) using the Admin Client

Performed on the server using the Admin Client for submission and job monitoring

Please review Job CLI for more information.

  1. Start the server service on Admin Server (nv-fl-aggreg)

    Expand to see an example starting the service on the Admin Server 
    $ cd ~/demo_env/workspace/demo01/prod_00/fl-junction.arcc.uwyo.edu/startup/
$ ./start.sh
(/home/<username>/nvflare_env) <username>@fl-junction:~/demo_env/workspace/demo01/prod_00/fl-junction.arcc.uwyo.edu/startup$ WORKSPACE set to /home/<username>/demo_env/workspace/demo01/prod_00/fl-junction.arcc.uwyo.edu/startup/..
PYTHONPATH is /local/custom:/home/<username>/NVFlare
start fl because of no pid.fl
new pid 3100
2025-12-02 09:52:51,859 - ServerDeployer - INFO - server heartbeat timeout set to 600
2025-12-02 09:52:51,930 - CoreCell - INFO - server: creating listener on grpc://0:8002
2025-12-02 09:52:52,037 - CoreCell - INFO - server: created backbone external listener for grpc://0:8002
2025-12-02 09:52:52,037 - CoreCell - INFO - server: creating listener on grpc://0:8003
2025-12-02 09:52:52,038 - CoreCell - INFO - server: created backbone external listener for grpc://0:8003
2025-12-02 09:52:52,038 - conn_manager - INFO - Connector [CH00003 PASSIVE tcp://0:4135] is starting
2025-12-02 09:52:52,539 - CoreCell - INFO - server: created backbone internal listener for tcp://localhost:4135
2025-12-02 09:52:52,539 - conn_manager - INFO - Connector [CH00001 PASSIVE grpc://0:8002] is starting
2025-12-02 09:52:52,539 - conn_manager - INFO - Connector [CH00002 PASSIVE grpc://0:8003] is starting
2025-12-02 09:52:52,543 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2025-12-02 09:52:52,544 - FederatedServer - INFO - max_reg_duration=60.0
2025-12-02 09:52:52,545 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Request
2025-12-02 09:52:52,545 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Abort
2025-12-02 09:52:52,545 - AuxRunner - INFO - registered aux handler for topic fed.event
2025-12-02 09:52:52,546 - ServerDeployer - INFO - deployed FLARE Server.
2025-12-02 09:52:52,547 - Server - INFO - added secure port at 0.0.0.0:8002
2025-12-02 09:52:52,547 - Server - INFO - added secure port at 0.0.0.0:8003
2025-12-02 09:52:52,549 - Cell - INFO - Register blob CB for channel='hci_channel', topic='*'
2025-12-02 09:52:52,549 - hci - INFO - Admin Server is started
2025-12-02 09:52:52,549 - root - INFO - Server started

  2. Start all Clients

    For this demonstration, we're using compute nodes on ARCC's Medicinebow Cluster

    1. Starting the first site/client - medbow1:

      Notice this is started from /project/arcc-nv-flare/software/demo_env/workspace/demo01/prod_00/medbow1/ on the Medicinebow system.

      Expand to show starting jobs on first client 
      [username@mblog1 ~]$ salloc -A arcc-nv-flare -c 8
salloc: Granted job allocation 48593887
salloc: Nodes t296 are ready for job
[]$
[]$ cd /project/arcc-nv-flare/software/demo_env/workspace/demo01/prod_00/medbow1/startup/
[]$ module load miniconda3
()[]$ conda activate /project/arcc-nv-flare/software/demo_client_env
()[]$ export PYTHONUSERBASE=$CONDA_PREFIX
()[]$ export PYTHONPATH=/project/arcc-nv-flare/software/NVFlare
()[]$ ./start.sh
(/project/arcc-nv-flare/software/demo_client_env) [username@t296 startup]$ WORKSPACE set to /project/arcc-nv-flare/software/demo_env/workspace/demo01/prod_00/medbow1/startup/..
PYTHONPATH is /local/custom:/project/arcc-nv-flare/software/NVFlare
start fl because of no pid.fl
new pid 3045922
2025-12-02 09:57:34,928 - CoreCell - INFO - medbow1: created backbone external connector to grpc://nv-fl-aggreg.arcc.uwyo.edu:8002
2025-12-02 09:57:34,930 - conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:31015] is starting
2025-12-02 09:57:35,273 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Request
2025-12-02 09:57:35,274 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Abort
2025-12-02 09:57:35,274 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2025-12-02 09:57:35,274 - AuxRunner - INFO - registered aux handler for topic fed.event
2025-12-02 09:57:35,275 - Communicator - INFO - Waiting for the client cell to be created.
2025-12-02 09:57:35,430 - CoreCell - INFO - medbow1: created backbone internal listener for tcp://localhost:31015
2025-12-02 09:57:35,431 - conn_manager - INFO - Connector [CH00001 ACTIVE grpc://nv-fl-aggreg.arcc.uwyo.edu:8002] is starting
2025-12-02 09:57:35,431 - Cell - INFO - Register blob CB for channel='server_command', topic='get_task'
2025-12-02 09:57:35,432 - Cell - INFO - Register blob CB for channel='server_command', topic='submit_update'
2025-12-02 09:57:35,432 - FederatedClient - INFO - Wait for engine to be created.
2025-12-02 09:57:35,448 - GrpcDriver - INFO - created secure channel at nv-fl-aggreg.arcc.uwyo.edu:8002
2025-12-02 09:57:35,449 - conn_manager - INFO - Connection [CN00002 N/A => nv-fl-aggreg.arcc.uwyo.edu:8002] is created: PID: 3045924
2025-12-02 09:57:35,826 - Authenticator - INFO - verified server identity 'nv-fl-aggreg.arcc.uwyo.edu'
2025-12-02 09:57:35,885 - Authenticator - INFO - Verified received token and signature successfully
2025-12-02 09:57:35,885 - FederatedClient - INFO - Successfully registered client:medbow1 for project demo01. Token:26941cb8-02d2-410b-bb06-4dcf379bc7f0 SSID:ebc6125d-0a56-4688-9b08-355fe9e4d61a
2025-12-02 09:57:35,890 - FederatedClient - INFO - Got engine after 0.4575202465057373 seconds
2025-12-02 09:57:35,890 - FederatedClient - INFO - Got the new primary SP: grpc://nv-fl-aggreg.arcc.uwyo.edu:8002

    2. Starting the second site/client - medbow2:

      Notice this is started from /project/arcc-nv-flare/software/demo_env/workspace/demo01/prod_00/medbow2/ on the Medicinebow system.

      Expand to show starting jobs on second client 
      # Exclude the compute node allocated about (it might be different to t296)
# This is so we have clients running oon separate compute nodes.
[salexan5@mblog2 ~]$ salloc -A arcc-nv-flare -c 8 --exclude=t296
salloc: Granted job allocation 48593888
salloc: Waiting for resource configuration
salloc: Nodes t439 are ready for job
[]$
[]$ cd /project/arcc-nv-flare/software/demo_env/workspace/demo01/prod_00/medbow2/startup/
[]$ module load miniconda3
[]$ conda activate /project/arcc-nv-flare/software/demo_client_env
() []$ export PYTHONUSERBASE=$CONDA_PREFIX
() []$ export PYTHONPATH=/project/arcc-nv-flare/software/NVFlare
() []$ ./start.sh
() []$ WORKSPACE set to /project/arcc-nv-flare/software/demo_env/workspace/demo01/prod_00/medbow2/startup/..
PYTHONPATH is /local/custom:/project/arcc-nv-flare/software/NVFlare
start fl because of no pid.fl
new pid 3046166
2025-12-02 09:58:28,904 - CoreCell - INFO - medbow2: created backbone external connector to grpc://fl-junction.arcc.uwyo.edu:8002
2025-12-02 09:58:28,906 - conn_manager - INFO - Connector [CH00002 PASSIVE tcp://0:22595] is starting
2025-12-02 09:58:29,340 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Request
2025-12-02 09:58:29,340 - AuxRunner - INFO - registered aux handler for topic ObjectStreamer.Abort
2025-12-02 09:58:29,341 - Cell - INFO - Register blob CB for channel='aux_communication', topic='*'
2025-12-02 09:58:29,341 - AuxRunner - INFO - registered aux handler for topic fed.event
2025-12-02 09:58:29,341 - Communicator - INFO - Waiting for the client cell to be created.
2025-12-02 09:58:29,407 - CoreCell - INFO - medbow2: created backbone internal listener for tcp://localhost:22595
2025-12-02 09:58:29,408 - conn_manager - INFO - Connector [CH00001 ACTIVE grpc://fl-junction.arcc.uwyo.edu:8002] is starting
2025-12-02 09:58:29,408 - Cell - INFO - Register blob CB for channel='server_command', topic='get_task'
2025-12-02 09:58:29,409 - Cell - INFO - Register blob CB for channel='server_command', topic='submit_update'
2025-12-02 09:58:29,409 - FederatedClient - INFO - Wait for engine to be created.
2025-12-02 09:58:29,424 - GrpcDriver - INFO - created secure channel at fl-junction.arcc.uwyo.edu:8002
2025-12-02 09:58:29,424 - conn_manager - INFO - Connection [CN00002 N/A => fl-junction.arcc.uwyo.edu:8002] is created: PID: 3046167
2025-12-02 09:58:29,863 - Authenticator - INFO - verified server identity 'fl-junction.arcc.uwyo.edu'
2025-12-02 09:58:29,932 - Authenticator - INFO - Verified received token and signature successfully
2025-12-02 09:58:29,933 - FederatedClient - INFO - Successfully registered client:medbow2 for project demo01. Token:d1080e3a-0db3-49f0-8773-b560f2695b18 SSID:ebc6125d-0a56-4688-9b08-355fe9e4d61a
2025-12-02 09:58:29,935 - FederatedClient - INFO - Got engine after 0.5260012149810791 seconds
2025-12-02 09:58:29,935 - FederatedClient - INFO - Got the new primary SP: grpc://fl-junction.arcc.uwyo.edu:8002

      Note that we can review and confirm on the server that the two clients have connected: 

      2025-12-02 09:52:53,045 - ServerState - INFO - Got the primary sp: fl-junction.arcc.uwyo.edu fl_port: 8002 SSID: ebc6125d-0a56-4688-9b08-355fe9e4d61a. Turning to hot.
2025-12-02 09:57:35,465 - conn_manager - INFO - Connection [CN00002 0.0.0.0:8002 <= ipv4:10.198.75.35:33770 SSL medbow1] is created: PID: 3101
2025-12-02 09:57:35,884 - ClientManager - INFO - Client: New client medbow1@10.198.75.35 joined. Sent token: 26941cb8-02d2-410b-bb06-4dcf379bc7f0.  Total clients: 1
2025-12-02 09:58:29,435 - conn_manager - INFO - Connection [CN00003 0.0.0.0:8002 <= ipv4:10.198.75.35:48052 SSL medbow2] is created: PID: 3101
2025-12-02 09:58:29,931 - ClientManager - INFO - Client: New client medbow2@10.198.75.35 joined. Sent token: d1080e3a-0db3-49f0-8773-b560f2695b18.  Total clients: 2

    3. Start the Admin Client (on the server)

      On nv-fl-aggreg: 
      $ cd ~/demo_env/workspace/demo01/prod_00/admin@uwyo.edu/startup
$ ./fl_admin.sh
User Name: admin@uwyo.edu
Connecting to FLARE ...
Trying to login, please wait ...
Type ? to list commands; type "? cmdName" to show usage of a command.
>
      On the server, we can review and confirm the admin client has also connected 
      2025-12-02 10:10:41,371 - conn_manager - INFO - Connection [CN00004 0.0.0.0:8003 <= ipv4:172.26.9.40:45154 SSL admin@uwyo.edu] is created: PID: 3101
2025-12-02 10:10:42,405 - ClientManager - INFO - Client: New admin admin@uwyo.edu@172.26.9.40 joined. Sent token: 06b70deb-1ac1-4eb1-a839-65947d8149d3.  Total clients: 2
2025-12-02 10:10:42,479 - LoginModule - INFO - Created user session for admin@uwyo.edu

    4. Check to confirm servers and clients are all running.

      Run the status check of your server and/or clients 
      > check_status server
Engine status: stopped
---------------------
| JOB_ID | APP NAME |
---------------------
---------------------
Registered clients: 2
--------------------------------------------------------------------------------------------------------
| CLIENT  | FQCN    | FQSN    | LEAF | TOKEN                                | LAST CONNECT TIME        |
--------------------------------------------------------------------------------------------------------
| medbow1 | medbow1 | medbow1 | True | 26941cb8-02d2-410b-bb06-4dcf379bc7f0 | Tue Dec  2 10:23:27 2025 |
| medbow2 | medbow2 | medbow2 | True | d1080e3a-0db3-49f0-8773-b560f2695b18 | Tue Dec  2 10:23:31 2025 |
--------------------------------------------------------------------------------------------------------          
Done [3700 usecs] 2025-12-02 10:23:34.141494

    5. Submit our job(s) using the Admin Client

      > submit_job /home/<username>/demo_job
Submitted job: 868d5662-8711-4b74-b148-b7907c1bfdd1
Done [74644 usecs] 2025-12-02 10:23:49.849712

    6. Monitor our Job(s) with the Admin Client and our Outputs 

      ## Run from Admin Client on nv-fl-aggreg VM
> list_jobs
----------------------------------------------------------------------------------------------------------------------------
| JOB ID                               | NAME     | STATUS             | SUBMIT TIME                      | RUN DURATION   |
----------------------------------------------------------------------------------------------------------------------------
| 868d5662-8711-4b74-b148-b7907c1bfdd1 | demo_job | RUNNING            | 2025-12-02T10:23:49.845926-07:00 | 0:00:01.623157 |
----------------------------------------------------------------------------------------------------------------------------
Done [5839 usecs] 2025-12-02 10:24:00.224126
      Expand to review initial submission information and output 
      ## nv-fl-aggreg
2025-12-02 10:23:50,002 - DefaultJobScheduler - INFO - [identity=server, run=?] - Try to schedule job 868d5662-8711-4b74-b148-b7907c1bfdd1, get result: (scheduled).
2025-12-02 10:23:50,003 - JobRunner - INFO - [identity=server, run=?] - Got the job: 868d5662-8711-4b74-b148-b7907c1bfdd1 from the scheduler to run
2025-12-02 10:23:50,026 - JobRunner - INFO - [identity=server, run=?] - Application app deployed to the server for job: 868d5662-8711-4b74-b148-b7907c1bfdd1
2025-12-02 10:23:50,027 - JobRunner - INFO - [identity=server, run=?] - App app to be deployed to the clients: medbow1,medbow2 for run: 868d5662-8711-4b74-b148-b7907c1bfdd1
2025-12-02 10:23:55,566 - JobRunner - INFO - [identity=server, run=?] - Updated the schedule history of Job: 868d5662-8711-4b74-b148-b7907c1bfdd1
2025-12-02 10:23:56,985 - ServerRunner - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - Server runner starting ...

## medbow1
2025-12-02 10:23:55,576 - ClientEngine - INFO - Starting client app. rank: 0
2025-12-02 10:23:55,595 - JobExecutor - INFO - Launched job 868d5662-8711-4b74-b148-b7907c1bfdd1 with job launcher: <class 'nvflare.app_common.job_launcher.client_process_launcher.ClientProcessJobLauncher'>
2025-12-02 10:24:02,722 - TaskScriptRunner - INFO - start task run() with full path: /project/arcc-nv-flare/software/demo_env/workspace/demo01/prod_00/medbow1/startup/../868d5662-8711-4b74-b148-b7907c1bfdd1/app_medbow1/custom/client.py

## medbow2
2025-12-02 10:23:55,576 - ClientEngine - INFO - Starting client app. rank: 0
2025-12-02 10:23:58,596 - JobExecutor - INFO - Launched job 868d5662-8711-4b74-b148-b7907c1bfdd1 with job launcher: <class 'nvflare.app_common.job_launcher.client_process_launcher.ClientProcessJobLauncher'>
2025-12-02 10:24:04,948 - TaskScriptRunner - INFO - start task run() with full path: /project/arcc-nv-flare/software/demo_env/workspace/demo01/prod_00/medbow2/startup/../868d5662-8711-4b74-b148-b7907c1bfdd1/app_medbow2/custom/client.py
      Expand to review processing output 
      ## nv-fl-aggreg
2025-12-02 10:23:56,986 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Beginning ScatterAndGather training phase.
2025-12-02 10:23:56,987 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Round 0 started.
2025-12-02 10:23:56,987 - WFCommServer - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - scheduled task train
2025-12-02 10:23:58,599 - JobRunner - INFO - [identity=server, run=?] - Started run: 868d5662-8711-4b74-b148-b7907c1bfdd1 for clients: medbow1,medbow2
2025-12-02 10:23:58,601 - JobRunner - INFO - [identity=server, run=?] - Job: 868d5662-8711-4b74-b148-b7907c1bfdd1 started to run, status changed to RUNNING.
2025-12-02 10:25:44,883 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller, peer=medbow1, peer_run=868d5662-8711-4b74-b148-b7907c1bfdd1, peer_rc=OK, task_name=train, task_id=edf826cc-7dd6-44d7-b5fc-d0fb657fcd20] - Contribution from medbow1 ACCEPTED by the aggregator at round 0.
2025-12-02 10:25:44,932 - ServerRunner - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller, peer=medbow1, peer_run=868d5662-8711-4b74-b148-b7907c1bfdd1, peer_rc=OK, task_name=train, task_id=edf826cc-7dd6-44d7-b5fc-d0fb657fcd20] - finished processing client result by controller
2025-12-02 10:25:44,932 - SubmitUpdateCommand - INFO - submit_update process. client_name:medbow1   task_id:edf826cc-7dd6-44d7-b5fc-d0fb657fcd20
2025-12-02 10:26:29,084 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller, peer=medbow2, peer_run=868d5662-8711-4b74-b148-b7907c1bfdd1, peer_rc=OK, task_name=train, task_id=2ed46c9f-d545-4f83-a3c2-c205adaf3131] - Contribution from medbow2 ACCEPTED by the aggregator at round 0.
2025-12-02 10:26:29,134 - ServerRunner - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller, peer=medbow2, peer_run=868d5662-8711-4b74-b148-b7907c1bfdd1, peer_rc=OK, task_name=train, task_id=2ed46c9f-d545-4f83-a3c2-c205adaf3131] - finished processing client result by controller
2025-12-02 10:26:29,134 - SubmitUpdateCommand - INFO - submit_update process. client_name:medbow2   task_id:2ed46c9f-d545-4f83-a3c2-c205adaf3131
2025-12-02 10:26:29,134 - WFCommServer - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - task train exit with status TaskCompletionStatus.OK
2025-12-02 10:26:29,134 - SubmitUpdateCommand - INFO - submit_update process. client_name:medbow2   task_id:2ed46c9f-d545-4f83-a3c2-c205adaf3131
2025-12-02 10:26:29,250 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Start aggregation.
2025-12-02 10:26:29,250 - DXOAggregator - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - aggregating 2 update(s) at round 0
2025-12-02 10:26:29,251 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - End aggregation.
2025-12-02 10:26:29,251 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Start persist model on server.
2025-12-02 10:26:29,253 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - End persist model on server.
2025-12-02 10:26:29,253 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Round 0 finished.
2025-12-02 10:26:29,301 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Round 1 started.

## medbow1
2025-12-02 10:24:02,722 - ClientRunner - INFO - [identity=medbow1, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - client runner started
2025-12-02 10:24:02,777 - Communicator - INFO - Received from server. getTask: train size: 251.9KB (251907 Bytes) time: 0.052593 seconds
2025-12-02 10:24:02,777 - FederatedClient - INFO - pull_task completed. Task name:train Status:True
2025-12-02 10:24:08,463 - TaskScriptRunner - INFO - site = medbow1, current_round=0
2025-12-02 10:24:57,204 - TaskScriptRunner - INFO - site=medbow1, Epoch: 0/2, Iteration: 3124, Loss: 320.46551513671875
2025-12-02 10:25:44,607 - TaskScriptRunner - INFO - site=medbow1, Epoch: 1/2, Iteration: 3124, Loss: 265.41162109375
2025-12-02 10:25:44,611 - TaskScriptRunner - INFO - Finished Training for medbow1
2025-12-02 10:25:44,627 - TaskScriptRunner - INFO - site: medbow1, sending model to server.
2025-12-02 10:25:44,934 - Communicator - INFO - SubmitUpdate to: server. size: 252KB (251975 Bytes). time: 0.072296 seconds
2025-12-02 10:25:44,934 - ClientRunner - INFO - [identity=medbow1, run=868d5662-8711-4b74-b148-b7907c1bfdd1, peer=demo01, peer_run=868d5662-8711-4b74-b148-b7907c1bfdd1, task_name=train, task_id=edf826cc-7dd6-44d7-b5fc-d0fb657fcd20] - task result sent to server
2025-12-02 10:26:31,081 - Communicator - INFO - Received from server. getTask: train size: 251.9KB (251907 Bytes) time: 0.010720 seconds
2025-12-02 10:26:31,081 - FederatedClient - INFO - pull_task completed. Task name:train Status:True

## medbow2
2025-12-02 10:24:04,949 - ClientRunner - INFO - [identity=medbow2, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - client runner started
2025-12-02 10:24:05,003 - Communicator - INFO - Received from server. getTask: train size: 251.9KB (251907 Bytes) time: 0.052134 seconds
2025-12-02 10:24:05,003 - FederatedClient - INFO - pull_task completed. Task name:train Status:True
2025-12-02 10:24:09,033 - TaskScriptRunner - INFO - site = medbow2, current_round=0
2025-12-02 10:26:04,020 - TaskScriptRunner - INFO - site=medbow2, Epoch: 0/2, Iteration: 3124, Loss: 320.8294372558594
2025-12-02 10:26:28,921 - TaskScriptRunner - INFO - site=medbow2, Epoch: 1/2, Iteration: 3124, Loss: 263.55560302734375
2025-12-02 10:26:28,923 - TaskScriptRunner - INFO - Finished Training for medbow2
2025-12-02 10:26:28,939 - TaskScriptRunner - INFO - site: medbow2, sending model to server.
2025-12-02 10:26:29,136 - Communicator - INFO - SubmitUpdate to: server. size: 252KB (251975 Bytes). time: 0.072612 seconds
2025-12-02 10:26:29,137 - ClientRunner - INFO - [identity=medbow2, run=868d5662-8711-4b74-b148-b7907c1bfdd1, peer=demo01, peer_run=868d5662-8711-4b74-b148-b7907c1bfdd1, task_name=train, task_id=2ed46c9f-d545-4f83-a3c2-c205adaf3131] - task result sent to server
2025-12-02 10:26:31,148 - Communicator - INFO - Received from server. getTask: train size: 251.9KB (251907 Bytes) time: 0.010979 seconds
2025-12-02 10:26:31,148 - FederatedClient - INFO - pull_task completed. Task name:train Status:True
      Expand to review initial submission information and output 
      ## nv-fl-aggreg
2025-12-02 10:33:39,670 - ScatterAndGather - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Finished ScatterAndGather Training.
2025-12-02 10:33:39,671 - ServerRunner - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Workflow: controller finalizing ...
2025-12-02 10:33:39,671 - ServerRunner - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - ABOUT_TO_END_RUN fired
2025-12-02 10:33:41,174 - ServerRunner - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Firing CHECK_END_RUN_READINESS ...
2025-12-02 10:33:41,282 - conn_manager - INFO - Connection [CN00006 Not Connected] is closed PID: 3101
2025-12-02 10:33:41,754 - conn_manager - INFO - Connection [CN00007 Not Connected] is closed PID: 3101
2025-12-02 10:33:43,182 - ServerRunner - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - END_RUN fired
2025-12-02 10:33:43,182 - ReliableMessage - INFO - ReliableMessage is shutdown
2025-12-02 10:33:43,182 - ServerRunner - INFO - [identity=demo01, run=868d5662-8711-4b74-b148-b7907c1bfdd1, wf=controller] - Server runner finished.
2025-12-02 10:33:45,053 - FederatedServer - INFO - Server app stopped.

## medbow1
2025-12-02 10:33:39,674 - ClientRunner - INFO - [identity=medbow1, run=868d5662-8711-4b74-b148-b7907c1bfdd1, peer=demo01, peer_run=868d5662-8711-4b74-b148-b7907c1bfdd1] - received request from Server to end current RUN
2025-12-02 10:33:40,869 - ClientRunner - INFO - [identity=medbow1, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - started end-run events sequence
2025-12-02 10:33:40,869 - ClientRunner - INFO - [identity=medbow1, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - ABOUT_TO_END_RUN fired
2025-12-02 10:33:40,869 - ClientRunner - INFO - [identity=medbow1, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - Firing CHECK_END_RUN_READINESS ...
2025-12-02 10:33:40,870 - InProcessClientAPI - WARNING - ask to stop job: reason: END_RUN received
2025-12-02 10:33:41,177 - InProcessClientAPI - WARNING - request to stop the job for reason END_RUN received
2025-12-02 10:33:41,179 - ClientRunner - INFO - [identity=medbow1, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - END_RUN fired
2025-12-02 10:33:41,184 - FederatedClient - INFO - Shutting down client run: medbow1
2025-12-02 10:33:42,116 - ClientRunner - INFO - [identity=medbow1, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - Client is stopping ...
2025-12-02 10:33:45,125 - MPM - INFO - MPM: Good Bye!
2025-12-02 10:33:45,751 - JobExecutor - INFO - run (868d5662-8711-4b74-b148-b7907c1bfdd1): child worker process finished with RC 0

## medbow2
2025-12-02 10:33:39,674 - ClientRunner - INFO - [identity=medbow2, run=868d5662-8711-4b74-b148-b7907c1bfdd1, peer=demo01, peer_run=868d5662-8711-4b74-b148-b7907c1bfdd1] - received request from Server to end current RUN
2025-12-02 10:33:41,488 - ClientRunner - INFO - [identity=medbow2, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - started end-run events sequence
2025-12-02 10:33:41,488 - ClientRunner - INFO - [identity=medbow2, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - ABOUT_TO_END_RUN fired
2025-12-02 10:33:41,489 - ClientRunner - INFO - [identity=medbow2, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - Firing CHECK_END_RUN_READINESS ...
2025-12-02 10:33:41,489 - InProcessClientAPI - WARNING - ask to stop job: reason: END_RUN received
2025-12-02 10:33:41,649 - InProcessClientAPI - WARNING - request to stop the job for reason END_RUN received
2025-12-02 10:33:41,650 - ClientRunner - INFO - [identity=medbow2, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - END_RUN fired
2025-12-02 10:33:41,656 - FederatedClient - INFO - Shutting down client run: medbow2
2025-12-02 10:33:41,854 - ClientRunner - INFO - [identity=medbow2, run=868d5662-8711-4b74-b148-b7907c1bfdd1] - Client is stopping ...
2025-12-02 10:33:44,862 - MPM - INFO - MPM: Good Bye!
2025-12-02 10:33:45,416 - JobExecutor - INFO - run (868d5662-8711-4b74-b148-b7907c1bfdd1): child worker process finished with RC 0

6. Shut down Client(s) and Server(s)

Every client must be shut down before shutting down server(s)

  1. Use the admin client to check what is currently running

    Expand to show the commands for reviewing what is currently running. 
    > check_status server
Engine status: stopped
---------------------
| JOB_ID | APP NAME |
---------------------
---------------------
Registered clients: 2
--------------------------------------------------------------------------------------------------------
| CLIENT  | FQCN    | FQSN    | LEAF | TOKEN                                | LAST CONNECT TIME        |
--------------------------------------------------------------------------------------------------------
| medbow1 | medbow1 | medbow1 | True | 1d24b61d-fa9f-4fb5-8cb4-b49d0d91e685 | Tue Dec  2 12:34:08 2025 |
| medbow2 | medbow2 | medbow2 | True | 774d732c-2120-47a7-9b6b-4325d5b9db03 | Tue Dec  2 12:33:57 2025 |
--------------------------------------------------------------------------------------------------------
Done [3576 usecs] 2025-12-02 12:34:13.065828

  2. Use the admin client to stop client(s)

    Expand to show the commands for reviewing what is currently running. 
    > shutdown client
Confirm with User Name: admin@uwyo.edu
------------------------------------
| CLIENT  | RESPONSE               |
------------------------------------
| medbow1 | Shutdown the client... |
| medbow2 | Shutdown the client... |
------------------------------------
Done [724497 usecs] 2025-12-02 12:34:46.595600
> check_status server
Engine status: stopped
---------------------
| JOB_ID | APP NAME |
---------------------
---------------------
Registered clients: 1
--------------------------------------------------------------------------------------------------------
| CLIENT  | FQCN    | FQSN    | LEAF | TOKEN                                | LAST CONNECT TIME        |
--------------------------------------------------------------------------------------------------------
| medbow2 | medbow2 | medbow2 | True | 774d732c-2120-47a7-9b6b-4325d5b9db03 | Tue Dec  2 12:33:57 2025 |
--------------------------------------------------------------------------------------------------------
Done [3718 usecs] 2025-12-02 12:34:51.135417
> shutdown client
Confirm with User Name: admin@uwyo.edu
------------------------------------
| CLIENT  | RESPONSE               |
------------------------------------
| medbow2 | Shutdown the client... |
------------------------------------
Done [659055 usecs] 2025-12-02 12:36:13.522850

  3. Use the admin client to shut down server(s)

    Expand to show the commands to shut down server. 
    > check_status server
Engine status: stopped
---------------------
| JOB_ID | APP NAME |
---------------------
---------------------
Registered clients: 0
Done [3297 usecs] 2025-12-02 12:36:18.036466
> shutdown server
Confirm with User Name: admin@uwyo.edu
Flare Server has been shutdown.
Done [3791 usecs] 2025-12-02 12:36:31.946861
Goodbye!

  4. Check for any lingering processes

    Expand to show commands to check for lingering processes. 
    medbow1
...
2025-12-02 12:34:45,875 - ClientEngine - INFO - Client shutdown...
About to shutdown the client...
2025-12-02 12:34:46,597 - FederatedClient - INFO - Shutting down client run: medbow1
2025-12-02 12:34:46,599 - FederatedClient - INFO - Logout client: medbow1 from server.
About to shutdown.
2025-12-02 12:34:50,594 - conn_manager - INFO - Connection [CN00002 Not Connected] is closed PID: 3084874
2025-12-02 12:34:50,722 - GrpcDriver - INFO - CLIENT: finished connection [CN00002 Not Connected]
2025-12-02 12:34:50,723 - MPM - INFO - MPM: Good Bye!
Process already terminated
# medbow2
...
2025-12-02 12:36:12,868 - ClientEngine - INFO - Client shutdown...
About to shutdown the client...
2025-12-02 12:36:13,523 - FederatedClient - INFO - Shutting down client run: medbow2
2025-12-02 12:36:13,527 - FederatedClient - INFO - Logout client: medbow2 from server.
2025-12-02 12:36:17,521 - conn_manager - INFO - Connection [CN00006 Not Connected] is closed PID: 3421803
2025-12-02 12:36:17,838 - GrpcDriver - INFO - CLIENT: finished connection [CN00006 Not Connected]
2025-12-02 12:36:17,839 - MPM - INFO - MPM: Good Bye!
# nv-fl-aggreg
...
2025-12-02 12:36:31,944 - ServerEngine - INFO - FL server shutdown.
2025-12-02 12:36:31,946 - JobRunner - INFO - [identity=server, run=?] - Stop all the running jobs.
2025-12-02 12:36:31,946 - FederatedServer - INFO - shutting down server
2025-12-02 12:36:31,946 - FederatedServer - INFO - canceling sync locks
2025-12-02 12:36:31,946 - FederatedServer - INFO - server off
2025-12-02 12:36:32,048 - conn_manager - INFO - Connection [CN00006 Not Connected] is closed PID: 7349
2025-12-02 12:36:32,380 - hci - INFO - Admin Server is stopped!
About to shutdown.
2025-12-02 12:36:35,882 - conn_manager - INFO - Connection [CN00012 Not Connected] is closed PID: 7349
2025-12-02 12:36:35,883 - MPM - INFO - MPM: Good Bye!
Process already terminated