Fast Monitoring System - PXI

From GlueXWiki
Jump to: navigation, search

Overview

A PXI system is built and configured to monitor the vital signals of the Hall-D super-conducting magnet to ensure its safe operation, particularly to detector the turn-to-turn short faults. Currently, the system contains 4 ADC modules which can sample up to 32 channels simultaneously at a maximum rate of 250 kHz. A LabVIEW RT program is automatically loaded and executed when the system is booted as a stand alone turn-key system. Through its Ethernet connection, the system hosts an EPICS server so that the measurements can be easily accessed and archived remotely. It also serves as an Ethernet device to the magnet control system based on PLC to provides various readings including current, field and voltages.

Ethernet Configuration

  • hostname: HallD-PXI.jlab.org
  • IP address: 129.57.37.80

Reboot the PXI system

Option I - local power cycling
  1. Climb the ladder and reach the top of the magnet where the PXI system is located.
  2. Reboot the PXI by pushing the power button in the front panel.
Option II - using remote PDU
  1. The power to the PXI system is distributed by a CyberPower PDU unit, the PXI can be rebooted using the PDU's web interface.
  2. Go to http://halldcppdu01.jlab.org (IP: 129.57.36.35).
  3. log in using user name halld and corresponding password (ask Elliott for password).
  4. In the left panel, click Outlet Control
  5. Check the #1 box, then choose Reboot in the pull down menu for the Select Task.
  6. Click Apply button to reboot the PXI
  7. One will be directed back to the Outlet Status page, and will see the status of PXI change to OFF then ON.
Option III - using a web browser

This method requires the web sever running properly on PXI.

  1. Go to http://halld-pxi.jlab.org (MS Silverlight plugin is required).
  2. Click Restart on the top right of the page to reboot the system remotely.
Option IV - using NI-MAX

This method requires the Ethernet interface running on PXI.

  1. Use remote desktop to log onto halld2.jlab.org as user halldusers. Ask Elliott for password.
  2. Double click the NI-MAX short-cut on the desktop to open Measurement & Automation Explorer.
  3. Expand the device tree on the left panel and select HallD-PXI under Remote Systems
  4. Click the Restart button on the right panel and acknowledge the prompt to reboot the system.

PXI DAQ Configuration and Channel List

By turning on the PXI system, a LabVIEW-RT program will be automatically executed and a DAQ session will be started. One can also stop a DAQ session and start a new one through EPICS control GUI.

During the initialization of a DAQ session, a configuration file Channel_Config.ini will be loaded, which defines various DAQ parameters as well as properties of individual channels.

A spreadsheet of the channel list is routinely updated and can be found in M drive:

M:\halld\controls\Rockwell Software\HallD_Controls_Development\Reference Documentation\PXI_Channel_List_date.xlsx

Access Channel_Config.ini

There are two methods to access and modify the file:

Option I - Web-Browser

The easiest way is to edit the file using a web-browser from a Windows machine (MS Silverlight plugin is required)

  1. Go to http://halld-pxi.jlab.org
  2. Click login on top-right of the page, type in user name 'halld' and password when prompted. Ask Elliott to get the password.
  3. Click the Remote File Browser from the left panel
  4. Expand the directory tree to open the folder called c:\ni-rt\Solenoid_DAQ
  5. Choose Channel_Config.ini in the right panel, then click Edit File ... to open it
  6. WARNING! Modify the file content if you know what you are doing, click Save Changes or Cancel to exit the editor.
Option II - FTP
  1. Use any ftp client to log on to halld-pxi.jlab.org as halld
  2. The configuration file can be found in directory /ni-rt/Solenoid_DAQ

Description of Channel_Config.ini

An example of the Channel_Config.ini is attached.

Some sections of the configuration are shown here with some description after #: 

[Global]
Sample_Rate = 100000   # ADC sampling rate in Hz, maximum for 4300 module is 250kHz
Report_Rate = 10       # EPICS report rate in Hz
Data_Rate = 5000       # ROOT file/local binary data file recording rate in Hz
Save_Length = 300      # Duration of a single local binary data file in seconds
Save_History = 0       # Number of binary data files kept on local disk
Fake_Signal = FALSE    # Flag for fake data stream

[Modules]              # Section for list of modules installed.
Slot2 = "4300"         # Module type in each slot.
Slot4 = "4300"         # Currently, only 4300 and 4300B are supported
Slot6 = "4300"         #
Slot8 = "4300B"        #

[Slot2_Ch0]            # Channel Section. The name has to follow the 'Slot#_Ch#' convention
Active = TRUE          # Channel activated for DAQ: TRUE or FALSE
Channel_Name = "VTT4"  # Channel name, will also be used for EPICS PV variables, eg: HallD-PXI:Data:VTT4 and HallD-PXI:Array:VTT4
Voltage_Range = 10     # Voltage range for ADC channel (+-), available ranges are: 1, 5 and 10 for 4300; 30, 150 and 300 for 4300B
Offset = -0.000169     # Offset applied for EPICS PV variables and PLC variables
Slope = 1              # Scaling factor applied for EPICS PV variables and PLC variables
Delay = 0              # Channel delay compensation in seconds

Communication with EPICS

NOTE: Make sure that gluon01.jlab.org(129.57.37.80) is included in EPICS_CA_ADDR_LIST

Types of EPICS variables

There are two types of EPICS variables served by this PXI system.

Single Elements

These variables are updated at the rate specified as Report_Rate in the configuration file. The PXI system average all readings between two reports to improve the data quality. The PV names for these variables are

HallD-PXI:Data:Channel_Name,

where the Channel_Name is defined in the configuration file.

Arrays

The arrays are mainly used for the remote archiving of the high speed data. Each array has a fixed dimension of 5000, and the update rate depends on the specified Data_Rate in the configuration. Same as the single element variables, these values are also taken from averages. The PV names for these variables are

HallD-PXI:Array:Channel_Name.

NOTE: In order to read these arrays, the EPICS array size limit needs to be modified by changing the value of EPICS_CA_MAX_ARRAY_BYTES to 100000

EPICS ProcServer for ROOT File Recording

An EPICS procserver is running on gluon01.jlab.org to save the high speed data to rootfiles when needed. All the rootfiles will be saved on gluon01 under directory

/local/scratch/PXI_DATA/ROOT

and period backed up on JLab silo tapes. To enable or disable the data writing, one can use the EPICS GUI

Reboot the ProcServer

  1. Log onto gluon01.jlab.org as user halld, ask Elliott for password,
  2. Connect to the PXI-EPICS server by typing telnet localhost 26060 or simply type rebootepics
  3. Hit Ctrl+X to reboot the server, reboot it a couple more times to make sure no error messages coming out,
  4. Hit Ctrl+] to exit the server
  5. Type 'quit' to exit telnet

NOTE: Best to REBOOT procserver if PXI system is restarted.

Communication with PLC

The PXI system communicates with the PLC magnet control as an Ethernet device.

Data from PXI to PLC
  • Port: 100
  • Format: Array of 36 integers in 32 bit (144 Bytes in total)
Data from PLC to PXI
  • Port: 200
  • Format: Array of 1 integers in 32 bit (4 Bytes in total)

A spread sheet is kept on M drive for the latest list of channels:

M:\halld\controls\Rockwell Software\HallD_Controls_Development\Reference Documentation\PXI_Tag_List_date.xlsx

How to Run PXI-EPICS Control

The PXI system can be controlled and monitored through a set of EPICS PVs. A user interface was created using Control System Studio (CSS).

List of Control/Status EPICS PVs

  • Control PVs
    • HallD-PXI:Control:Start: start DAQ
    • HallD-PXI:Control:Stop: stop DAQ
    • HallD-PXI:Control:Quit: quit the LabVIEW-RT program
    • HallD-PXI:Control:Zero: take 1-minuite data to calculate offsets of all channels
    • HallD-PXI:Control:Reset: reset offsets of all channels
    • HallD-PXI:Control:ForceAlarm: force the turn-to-turn alarm to be true
  • Status PVs
    • HallD-PXI:Status:Beat: heart beat, updating at 1Hz
    • HallD-PXI:Status:Status: indicator of current machine state
    • HallD-PXI:Status:Error: error bit of the LabVIEW-RT program
    • HallD-PXI:Status:Message: error message of the LabVIEW-RT program
    • HallD-PXI:Status:CPU: CPU usage of the PXI system
    • HallD-PXI:Status:Interval: Time interval between two EPICS updates
    • HallD-PXI:Status:Jitter: Jitter of time intervals
    • HallD-PXI:Status:FIFO: Buffer depth for local data writing, depleted when reaches 1
    • HallD-PXI:Status:Alarm: turn-to-turn short alarm bit

Start EPICS Control

  1. Log onto gluon01.jlab.org using username halld from any Linux terminal, ask Elliott if you do not know the password.
  2. Type css & to start the Control System Studio.
  3. If PXI control is not already loaded, click File -> Open File in the top left menu, open ~halld/CSS-Workspaces/Default/Hall-D/PXI/PXI_Status.opi.

Use EPICS Control

NOTE: Make sure that the PXI's heart beat is blinking, and all the EPICS channels are connected.

Main Screen

The main screen gives you the access to PXI control, ROOT File Recording, strip-charts for CPU usage, solenoid current, field strength, pickup coil signals and buttons to open other strip-charts for sub coil voltages and voltage ratios.

To Start DAQ

  1. By default, the PXI-DAQ should be running, if not, hit Start to start the DAQ process, you should see that the status changes from System Standby to Initialize DAQ to DAQ Running

Calculate Offsets

  1. Make sure magnet is in a state that all signals remain zero,
  2. Stop running DAQ proccess by hitting Stop,
  3. Click zero button to start zeroing process, the status will change from System Standby to Initizalize to Zeroing. After about 1 minute, the zeroing process should automatically stop and System Standby should be shown then.
  4. One needs to start a normal DAQ afterwards.

Reset Offsets

  1. Stop DAQ
  2. Click Reset Offset button
  3. Start DAQ

Generate Fake Turn-to-Turn Short Alarm

The feature only works during a normal DAQ process

  1. Click T-2-T Alarm to trigger an alarm
  2. Click T-2-T Alarm again to return to disable the fake alarm

Open Strip Charts

Individual Voltage Taps
  1. Click Open Sub Coil Chart
Total Voltage and Ratios of Voltages
  1. Click Open Ratio Chart button
Quench Detector and Voltage Leads Signals
  1. Click Open QD/Leads Chart button

Start or Stop ROOT File Recording

  1. ROOT Files need to be recorded for offline analysis, start a new ROOT file everytime a new ramp up starts from 0 A.
  2. Check the status of ROOT File Recording, if it says Idle .. Click to start new file, then click the button, acknowledge the popup message to start a new ROOT file. The button should then say Writing Data .. Click to stop.
  3. Double check the existance of the new file.
    1. first log onto gluon01 from a Linux terminal,
    2. then type ll -h /local/scratch/PXI_DATA/ROOT/*.root or check to display the list of ROOT files, try to find the file with the latest time stamp, also repeat the command a couple of more times to make sure that the file size is increasing.
  4. If the ROOT file is neither found nor being updated, one needs to restart the EPICS ProcService.
  5. Stop ROOT file writing by clicking the button after the current ramps down to 0 A.
  6. Add the new file to the file list in elog

How to Analyize ROOT Files

List of ROOT Files

  • HallD Elog entry of list of ROOT files
  • all the ROOT files are on gluon01.jlab.org in directory /local/scratch/PXI_DATA/ROOT, there is also a symbolic link ./data in the precompiled Analyzer directory on gluon01.

Structure of ROOT Files

Channels in ROOT Files
  • b_field: Magnetic field reading from Hall probe
  • i_shunt: Voltage drop across the shunt resistor which is proportional to the current
  • i_sol: Magnetic current reading from ZFCT unit
  • qd_err: Error signal from the Quench Detector
  • v_mps: Total voltage from the magnet power supply (MPS)
  • v_pc1u - v_pc4d: voltages from pickup coils
  • vtt3 - vtt21: voltages across voltage taps
Raw Data Files
  • As stated as raw, all the readings in these ROOT files does not include calibrations such as baseline adjustment and scaling factor.
  • Data from different channels are saved in separate trees named as halld-pxi:array:Channel_Name.
  • Each tree contains a single branch called b_record.
  • The branch consists of the following leaves:
    • record_tsec: Long64_4, the second of the timestamp
    • record_tnes: Long64_4, the nanosecond of the timestamp
    • record_data: Float_t[5000], a pack of 5000 readings
Resampled Data Files
  • Data files with reduced sampling rates can be generated using the ROOT File Analyzer.
  • The readings include calibrations such as baseline adjustment and scaling factors.
  • Each file contains a single tree called T.
  • The tree consists of the following branches:
    • time: Double_t, time elapsed since the DAQ starts
    • Channel_Name: Float_t, readings of individual channels

Solenoid ROOT File Analysis Tool

Run the Analyzer

  1. Log onto gluon01 using username halld: ssh halld@gluon01
  2. Go to analyzer directory
    • cd /home/halld/solenoid/Analyzer
    • Or simply type gowork
  3. Run the analyzer: ./analyzer

Checkout and Compile the Analyzer from Source Code

The source files of the Analyzer are stored in the Hall-D SVN repository.

  1. Check out the code
  2. Make sure that the ROOT library is included in LD_LIBRARY_PATH environment variable
  3. Compile the code: make
  4. Execute the compiled binary: ./analyzer

Use the Analyzer

The Analyzer's GUI is split into one top panel and one bottom panel. The top panel shows the magnet current history and the bottom panel is used to show plots for different analysis.

NOTE: The horizontal range of the current history plot is used to control the data range for all the analysis.

Usage of Buttons and Fields

Open
Open a ROOT file, the file can either be a raw data file created by the EPICS ProcServer, or a resampled data file created by the Analyzer.
Calculate (Offset)
Calculate offsets of all channels in selected range
Save (Offset)
Save calculated offset values into a txt file
Load (Offset)
Load offset values from a txt file
Reset (Offset)
Reset all offsets to zero
Exit
Exit the Analyzer
Hall Probe Delay
Delay time (s) for the Hall probe readout, put 0 unless instructed by experts
Resample Rreq.
Resample frequency (Hz) for plots, can be 0 - 5 kHz, the higher the slower
dI/dt Windows
Windows size (s) used to calculate the current ramp rate
Legend
Show legends in plots
Mean
Calculate mean values of individual histograms shown in the bottom panel, the results are printed in terminal
Fit
Do 2nd order polynomial fits on inductance, the results are printed in terminal as A B C where y = A + Bx +Cx2
Save ROOT File
Create a ROOT file with reduced sampling rate and offsets, the created file can be found in directory ./rootfiles
Field/Current
Plot current, magnetic field measured by the Hall probe and the ratio. The ratio can be used to identify turn-to-turn short.
Pickup Coils
Plot pickup coil signals, field calculated by integrating pickup coil outputs, and ratio of field to current. The ratio can be used to identify turn-to-turn short.
Voltage
Plot voltage reading of all sub-coils as well as the total voltages
Quench Detector
(only available in new version) Plot Quench detector error signal and voltage across leads
Inductance
Plot current change rate, inductance of sub-coils calculated based on L = V/(dI/dt) as a function of current.
Resistance
Plot current change rate and resistance calculated based on R = (V-L*dI/dt)/I
Temperature
Plot magnet current and temperature of sub-coils estimated based on their resistance values
Print
Print the bottom panel into a PDF file and a PNG file, the image files can be found in directory ./plots
Retrieved from "https://halldweb.jlab.org/wiki/index.php?title=Fast_Monitoring_System_-_PXI&oldid=64820"