fi_poll_open

A poll set is defined with the following attributes.

struct fi_poll_attr {
    uint64_t             flags;     /* operation flags */
};

    

flags

Flags that set the default operation of the poll set. The use of this field is reserved and must be set to 0 by the caller.

fi_close

fi_poll_add

fi_poll_del

fi_poll

Note that fi_poll only indicates that events might be available. In some cases, providers may consume such events internally, to drive progress, for example. This can result in fi_poll returning false positives. Applications should drive their progress based on the results of reading events from a completion queue or reading counter values. The fi_poll function will always return all completion queues and counters that do have new events.

fi_wait_open

The properties and behavior of a wait set are defined by struct fi_wait_attr.

struct fi_wait_attr {
    enum fi_wait_obj     wait_obj;  /* requested wait object */
    uint64_t             flags;     /* operation flags */
};

    

wait_obj

Wait sets are associated with specific wait object(s). Wait objects allow applications to block until the wait object is signaled, indicating that an event is available to be read. The following values may be used to specify the type of wait object associated with a wait set: FI_WAIT_UNSPEC, FI_WAIT_FD, FI_WAIT_MUTEX_COND, and FI_WAIT_YIELD.

- FI_WAIT_UNSPEC

Specifies that the user will only wait on the wait set using fabric interface calls, such as fi_wait. In this case, the underlying provider may select the most appropriate or highest performing wait object available, including custom wait mechanisms. Applications that select FI_WAIT_UNSPEC are not guaranteed to retrieve the underlying wait object.

- FI_WAIT_FD

Indicates that the wait set should use a single file descriptor as its wait mechanism, as exposed to the application. Internally, this may require the use of epoll in order to support waiting on a single file descriptor. File descriptor wait objects must be usable in the POSIX select(2) and poll(2), and Linux epoll(7) routines (if available). Provider signal an FD wait object by marking it as readable or with an error.

- FI_WAIT_MUTEX_COND

Specifies that the wait set should use a pthread mutex and cond variable as a wait object.

- FI_WAIT_POLLFD

This option is similar to FI_WAIT_FD, but allows the wait mechanism to use multiple file descriptors as its wait mechanism, as viewed by the application. The use of FI_WAIT_POLLFD can eliminate the need to use epoll to abstract away needing to check multiple file descriptors when waiting for events. The file descriptors must be usable in the POSIX select(2) and poll(2) routines, and match directly to being used with poll. See the NOTES section below for details on using pollfd.

- FI_WAIT_YIELD

Indicates that the wait set will wait without a wait object but instead yield on every wait.

flags

Flags that set the default operation of the wait set. The use of this field is reserved and must be set to 0 by the caller.

fi_close

fi_wait

fi_trywait

The fi_trywait() call is used in conjunction with native operating system calls to block on wait objects, such as file descriptors. The application must call fi_trywait and obtain a return value of FI_SUCCESS prior to blocking on a native wait object. Failure to do so may result in the wait object not being signaled, and the application not observing the desired events. The following pseudo-code demonstrates the use of fi_trywait in conjunction with the OS select(2) call.

fi_control(&cq->fid, FI_GETWAIT, (void *) &fd);
FD_ZERO(&fds);
FD_SET(fd, &fds);
while (1) {
    if (fi_trywait(&cq, 1) == FI_SUCCESS)
        select(fd + 1, &fds, NULL, &fds, &timeout);
    do {
        ret = fi_cq_read(cq, &comp, 1);
    } while (ret > 0);
}

    

fi_trywait() will return FI_SUCCESS if it is safe to block on the wait object(s) corresponding to the fabric descriptor(s), or -FI_EAGAIN if there are events queued on the fabric descriptor or if blocking could hang the application.

The call takes an array of fabric descriptors. For each wait object that will be passed to the native wait routine, the corresponding fabric descriptor should first be passed to fi_trywait. All fabric descriptors passed into a single fi_trywait call must make use of the same underlying wait object type.

The following types of fabric descriptors may be passed into fi_trywait: event queues, completion queues, counters, and wait sets. Applications that wish to use native wait calls should select specific wait objects when allocating such resources. For example, by setting the item’s creation attribute wait_obj value to FI_WAIT_FD.

In the case the wait object to check belongs to a wait set, only the wait set itself needs to be passed into fi_trywait. The fabric resources associated with the wait set do not.

On receiving a return value of -FI_EAGAIN from fi_trywait, an application should read all queued completions and events, and call fi_trywait again before attempting to block. Applications can make use of a fabric poll set to identify completion queues and counters that may require processing.

fi_control

FI_GETWAIT (void **)

This command allows the user to retrieve the low-level wait object associated with a wait set or fid. The format of the wait set is specified during wait set creation, through the wait set attributes. The fi_control arg parameter should be an address where a pointer to the returned wait object will be written. This should be an ’int *’ for FI_WAIT_FD, `struct fi_mutex_cond' for FI_WAIT_MUTEX_COND, or `struct fi_wait_pollfd' for FI_WAIT_POLLFD. Support for FI_GETWAIT is provider specific.

FI_GETWAITOBJ (enum fi_wait_obj *)

This command returns the type of wait object associated with a wait set or fid.