Download as pdf or txt
Download as pdf or txt
You are on page 1of 36

LinuxPCIdrivers

Linux
PCIdrivers
MichaelOpdenacker
FreeElectrons

http://en.wikipedia.org/wiki/Image:PCI_Slots_Digon3.JPG

Copyright20042009,FreeElectrons.
CreativeCommonsBYSA3.0license
Latestupdate:Jul13,2010,
Documentsources,updatesandtranslations:
http://freeelectrons.com/docs/pcidrivers
Corrections,suggestions,contributionsandtranslationsarewelcome!

1
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

LinuxPCIdrivers

UnderstandingPCI

2
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIbusfamily
ThefollowingbusesbelongtothePCIfamily:
PCI
32bitbus,33or66MHz
MiniPCI
Smallerslotinlaptops
CardBus
Externalcardslotinlaptops
PIXExtended(PCIX)
WiderslotthanPCI,64bit,but
canacceptastandardPCIcard

PCIExpress(PCIeorPCIE)
CurrentgenerationofPCI.Serial
insteadofparallel.
PCIExpressMiniCard
ReplacesMiniPCIinrecent
laptops
ExpressCard
ReplacesCardBusinrecent
laptops

Thesetechnologiesarecompatibleandcanbehandledbythesamekerneldrivers.
Thekerneldoesn'tneedtoknowwhichexactslotandbusvariantisused.

3
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIdevicetypes
MaintypesofdevicesfoundonthePCIbus
Networkcards(wiredorwireless)
SCSIadapters
Buscontrollers:USB,PCMCIA,I2C,FireWire,IDE
Graphicsandvideocards
Soundcards

4
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIfeatures
Fordevicedriverdevelopers
Deviceresources(I/Oaddresses,IRQlines)automatically
assignedatboottime,eitherbytheBIOSorbyLinuxitself(if
configured).
Thedevicedriverjusthastoreadthecorresponding
configurationssomewhereinthesystemaddressspace.
Endianism:PCIdeviceconfigurationinformationisLittleEndian.
Rememberthatinyourdrivers
(conversiontakencareofbysomekernelfunctions).

5
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIdevicelist(1)
lscpi
00:00.0Hostbridge:IntelCorporationMobile945GM/PM/GMS,943/940GMLand945GTExpressMemoryControllerHub(rev03)
00:02.0VGAcompatiblecontroller:IntelCorporationMobile945GM/GMS,943/940GMLExpressIntegratedGraphicsController(rev03)
00:02.1Displaycontroller:IntelCorporationMobile945GM/GMS/GME,943/940GMLExpressIntegratedGraphicsController(rev03)
00:1b.0Audiodevice:IntelCorporation82801G(ICH7Family)HighDefinitionAudioController(rev01)
00:1c.0PCIbridge:IntelCorporation82801G(ICH7Family)PCIExpressPort1(rev01)
00:1c.1PCIbridge:IntelCorporation82801G(ICH7Family)PCIExpressPort2(rev01)
00:1c.2PCIbridge:IntelCorporation82801G(ICH7Family)PCIExpressPort3(rev01)
00:1d.0USBController[...]
00:1e.0PCIbridge:IntelCorporation82801MobilePCIBridge(reve1)
00:1f.0ISAbridge:IntelCorporation82801GBM(ICH7M)LPCInterfaceBridge(rev01)
00:1f.1IDEinterface:IntelCorporation82801G(ICH7Family)IDEController(rev01)
00:1f.3SMBus:IntelCorporation82801G(ICH7Family)SMBusController(rev01)
02:01.0CardBusbridge:RicohCoLtdRL5c476II(revb4)
02:01.1FireWire(IEEE1394):RicohCoLtdR5C552IEEE1394Controller(rev09)
02:01.2SDHostcontroller:RicohCoLtdR5C822SD/SDIO/MMC/MS/MSProHostAdapter(rev18)
09:00.0Ethernetcontroller:BroadcomCorporationNetXtremeBCM5752GigabitEthernetPCIExpress(rev02)
0c:00.0Networkcontroller:IntelCorporationPRO/Wireless4965AGorAGNNetworkConnection(rev61)
lspcitv
[0000:00]+00.0IntelCorporationMobile945GM/PM/GMS,943/940GMLand945GTExpressMemoryControllerHub
+02.0IntelCorporationMobile945GM/GMS,943/940GMLExpressIntegratedGraphicsController
+02.1IntelCorporationMobile945GM/GMS/GME,943/940GMLExpressIntegratedGraphicsController
+1b.0IntelCorporation82801G(ICH7Family)HighDefinitionAudioController
+1c.0[0000:0b]
+1c.1[0000:0c]00.0IntelCorporationPRO/Wireless4965AGorAGNNetworkConnection
+1c.2[0000:09]00.0BroadcomCorporationNetXtremeBCM5752GigabitEthernetPCIExpress
+1d.0IntelCorporation82801G(ICH7Family)USBUHCIController#1[...]
+1e.0[0000:0206]+01.0RicohCoLtdRL5c476II
|+01.1RicohCoLtdR5C552IEEE1394Controller
|\01.2RicohCoLtdR5C822SD/SDIO/MMC/MS/MSProHostAdapter
+1f.0IntelCorporation82801GBM(ICH7M)LPCInterfaceBridge
+1f.1IntelCorporation82801G(ICH7Family)IDEController
\1f.3IntelCorporation82801G(ICH7Family)SMBusController

6
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIdevicelist(2)
lspcienumeratesallPCIdevices
02:01.0CardBusbridge:RicohCoLtdRL5c476II(revb4)
02:01.1FireWire(IEEE1394):RicohCoLtdR5C552IEEE1394Controller
02:01.2SDHostcontroller:RicohCoLtdR5C822SD/SDIO/MMC/MS/MSPro

Functionnumber
PCIdevicenumber
PCIbusnumber

lscpitshowsthebusdevicetree
[0000:00]+00.0IntelCorporationMobile945GM/PM/GMS,943/940GMLand945GTExpressMemoryControllerHub
+02.0IntelCorporationMobile945GM/GMS,943/940GMLExpressIntegratedGraphicsController
+02.1IntelCorporationMobile945GM/GMS/GME,943/940GMLExpressIntegratedGraphicsController
+1b.0IntelCorporation82801G(ICH7Family)HighDefinitionAudioController
+1c.0[0000:0b]
+1c.1[0000:0c]00.0IntelCorporationPRO/Wireless4965AGorAGNNetworkConnection
+1c.2[0000:09]00.0BroadcomCorporationNetXtremeBCM5752GigabitEthernetPCIExpress
+1d.0IntelCorporation82801G(ICH7Family)USBUHCIController#1[...]
+1e.0[0000:0206]+01.0RicohCoLtdRL5c476II
|+01.1RicohCoLtdR5C552IEEE1394Controller
|\01.2RicohCoLtdR5C822SD/SDIO/MMC/MS/MSProHostAdapter

PCIdomain
PCIbus0

PCIbridge PCIbus2
7
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIdevicelist(3)
Thistreestructurereflectsthestructurein/sys:
/sys/devices/pci0000:00/0000:00:1e.0/0000:02:01.2

Bus0

PCIbridge

Bus2

8
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIdeviceconfiguration(1)
EachPCIdevicehasa256byteaddressspacecontaining
configurationregisters.
Deviceconfigurationcanbedisplayedwithlspcix:
0c:00.0Networkcontroller:IntelCorporationPRO/Wireless
4965AGorAGNNetworkConnection(rev61)
00:86802942060410006100800210000000
10:04e0dfef000000000000000000000000
20:00000000000000000000000086802111
30:00000000c80000000000000005010000

9
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIdeviceconfiguration(2)
StandardinformationfoundinPCIconfigurations:
Offset0:VendorId
Offset2:DeviceId
Offset10:ClassId(network,display,bridge...)
Offsets16to39:BaseAddressRegisters(BAR)0to5
Offset44:SubvendorId
Offset46:SubdeviceId
Offsets64andup:uptothedevicemanufacturer
Kernelsources:theseoffsetsaredefinedin
include/linux/pci_regs.h
10
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

LinuxPCIdrivers

ImplementingLinuxdrivers

11
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Registeringsupporteddevices
Fromdrivers/net/ne2kpci.c(Linux2.6.27):
staticstructpci_device_idne2k_pci_tbl[]={
{0x10ec,0x8029,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_RealTek_RTL_8029},
{0x1050,0x0940,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_Winbond_89C940},
{0x11f6,0x1401,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_Compex_RL2000},
{0x8e2e,0x3000,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_KTI_ET32P2},
{0x4a14,0x5000,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_NetVin_NV5000SC},
{0x1106,0x0926,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_Via_86C926},
{0x10bd,0x0e34,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_SureCom_NE34},
{0x1050,0x5a5a,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_Winbond_W89C940F},
{0x12c3,0x0058,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_Holtek_HT80232},
{0x12c3,0x5598,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_Holtek_HT80229},
{0x8c4a,0x1980,PCI_ANY_ID,PCI_ANY_ID,0,0,CH_Winbond_89C940_8c4a},
{0,}
};
MODULE_DEVICE_TABLE(pci,ne2k_pci_tbl);

12
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Registeringadriver(1)
Declaringdriverhooksandsupporteddevicestable:
staticstructpci_driverne2k_driver={
.name=DRV_NAME,
.probe=ne2k_pci_init_one,
.remove=__devexit_p(ne2k_pci_remove_one),
.id_table=ne2k_pci_tbl,
#ifdefCONFIG_PM
.suspend=ne2k_pci_suspend,
.resume=ne2k_pci_resume,
#endif/*CONFIG_PM*/
};

13
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Registeringadriver(2)
staticint__initne2k_pci_init(void)
{
returnpci_register_driver(&ne2k_driver);
}
staticvoid__exitne2k_pci_cleanup(void)
{
pci_unregister_driver(&ne2k_driver);
}
Thehooksandsupporteddevicesareloadedatmoduleloading
time.
Theprobe()hookgetscalledbythePCIgenericcode
whenamatchingdeviceisfound.
VerysimilartoUSBdevicedrivers!
14
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Markingdriverhooks(1)
__init:moduleinitfunction.
Codediscardedafterdriverinitialization.
__exit:moduleexitfunction.
Ignoredforstaticallycompileddrivers.
__devinit:probefunctionandallinitializationfunctions
NormalfunctionifCONFIG_HOTPLUGisset.Identicalto__init
otherwise.
__devinitconst:forthedeviceidtable

15
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Markingdriverhooks(2)
__devexit:functionscalledatremovetime.
Samecaseasin__devinit
Allreferencesto__devinitfunctionaddressesshouldbe
declaredwith__devexit_p(fun).Thisreplacesthe
functionaddressbyNULLifthiscodeisdiscarded.
Example:samedriver:
staticstructpci_driverne2k_driver={
.name=DRV_NAME,
.probe=ne2k_pci_init_one,
.remove=
__devexit_p(ne2k_pci_remove_one),
.id_table=ne2k_pci_tbl,
...
};

16
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Deviceinitializationsteps
Enablethedevice
RequestI/OportandI/Omemoryresources
SettheDMAmasksize
(forbothcoherentandstreamingDMA)
Allocateandinitializesharedcontroldata
(pci_allocate_coherent())
Initializedeviceregisters(ifneeded)
RegisterIRQhandler(request_irq())
Registertoothersubsystems(network,video,disk,etc.)
EnableDMA/processingengines.
17
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Enablingthedevice
Beforetouchinganydeviceregisters,thedrivershouldfirst
executepci_enable_device().Thiswill:
Wakeupthedeviceifitwasinsuspendedstate
AllocateI/Oandmemoryregionsofthedevice
(ifnotalreadydonebytheBIOS)
AssignanIRQtothedevice
(ifnotalreadydonebytheBIOS)
pci_enable_device()canfail.Checkthereturnvalue!

18
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

pci_enable_deviceexample
Fromdrivers/net/ne2kpci.c(Linux2.6.27):
staticint__devinitne2k_pci_init_one
(structpci_dev*pdev,
conststructpci_device_id*ent)
{
...
i=pci_enable_device(pdev);
if(i)
returni;
...
}

19
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Enablingthedevice(2)
EnableDMAbycallingpci_set_master().Thiswill:
EnableDMAbysettingthebusmasterbitinthe
PCI_COMMANDregister.Thedevicewillthenbeabletoact
asamasterontheaddressbus.
Fixthelatencytimervalueifit'ssettosomethingbogusby
theBIOS.
IfthedevicecanusethePCIMemoryWriteInvalidate
transaction(writingentirecachelines),youcanalsocall
pci_set_mwi():
ThisenablesthePCI_COMMANDbitforMemoryWrite
Invalidate
Thisalsoensuresthatthecachelinesizeregisterisset
correctly.

20

FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Accessingconfigurationregisters(1)
NeededtoaccessI/Omemoryandportinformation
#include<linux/pci.h>

Reading:
intpci_read_config_byte(structpci_dev*dev,
intwhere,u8*val);
intpci_read_config_word(structpci_dev*dev,
intwhere,u16*val);
intpci_read_config_dword(structpci_dev*dev,
intwhere,u32*val);

Example:drivers/net/cassini.c
pci_read_config_word(cp>pdev,PCI_STATUS,&cfg);

21
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Accessingconfigurationregisters(2)
Writing:
intpci_write_config_byte(structpci_dev*dev,
intwhere,u8val);
intpci_write_config_word(structpci_dev*dev,
intwhere,u16val);
intpci_write_config_dword(structpci_dev*dev,
intwhere,u32val);

Example:drivers/net/s2io.c
/*Clear"detectedparityerror"bit
pci_write_config_word(sp>pdev,PCI_STATUS,0x8000);

22
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

AccessingI/Oregistersandmemory(1)
EachPCIdevicecanhaveupto6I/Oormemoryregions,
describedinBAR0toBAR5.
AccessthebaseaddressoftheI/Oregion:
#include<linux/pci.h>
longiobase=pci_resource_start(pdev,bar);
AccesstheI/Oregionsize:
longiosize=pci_resource_len(pdev,bar);
ReservetheI/Oregion:
request_region(iobase,iosize,mydriver);
orsimpler:
pci_request_region(pdev,bar,mydriver);
orevensimpler(regionsforallBARs):
pci_request_regions(pdev,mydriver);

23
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

AccessingI/Oregistersandmemory(2)
Fromdrivers/net/ne2kpci.c(Linux2.6.27):
ioaddr=pci_resource_start(pdev,0);
irq=pdev>irq;
if(!ioaddr||((pci_resource_flags(pdev,0)&IORESOURCE_IO)==0))
{
dev_err(&pdev>dev,"noI/OresourceatPCIBAR#0\n");
returnENODEV;
}
if(request_region(ioaddr,NE_IO_EXTENT,DRV_NAME)==NULL){
dev_err(&pdev>dev,"I/Oresource0x%x@0x%lxbusy\n",
NE_IO_EXTENT,ioaddr);
returnEBUSY;
}

24
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

SettingtheDMAmasksize(1)
Usepci_dma_set_mask()todeclareanydevicewithmore(or
less)than32bitbusmastercapability
Inparticular,mustbedonebydriversforPCIXandPCIe
compliantdevices,whichuse64bitDMA.
Ifthedevicecandirectlyaddress"consistentmemory"inSystem
RAMabove4Gphysicaladdress,registerthisbycalling
pci_set_consistent_dma_mask().

25
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

SettingtheDMAmasksize(2)
Example(drivers/net/wireless/ipw2200.cinLinux2.6.27):
err=pci_set_dma_mask(pdev,DMA_32BIT_MASK);
if(!err)
err=pci_set_consistent_dma_mask(pdev,DMA_32BIT_MASK);
if(err){
printk(KERN_WARNINGDRV_NAME":NosuitableDMAavailable.\n");
gotoout_pci_disable_device;
}

26
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

AllocateconsistentDMAbuffers
NowthattheDMAmasksizehasbeenallocated...
Youcanallocateyourcacheconsistentbuffers
ifyouplantousesuchbuffers.
SeeourDMApresentation
andDocumentation/DMAAPI.txtfordetails.

27
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Initializedeviceregisters
Ifneededbythedevice
Setsomecapabilityfields
Dosomevendorspecificinitializationorreset
Example:clearpendinginterrupts.

28
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Registerinterrupthandlers
Needtocallrequest_irq()withtheIRQF_SHAREDflag,
becauseallPCIIRQlinescanbeshared.
Registrationalsoenablesinterrupts,soatthispoint
Makesurethatthedeviceisfullyinitializedandreadyto
serviceinterrupts.
Makesurethatthedevicedoesn'thaveanypending
interruptbeforecallingrequest_irq().
Whereyouactuallycallrequest_irq()canactually
dependonthetypeofdeviceandthesubsystemitcouldbe
partof(network,video,storage...).
Yourdriverwillthenhavetoregistertothissubsystem.
29
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIdeviceshutdown(1)
Intheremove()function,youtypicallyhavetoundo
whatyoudidatdeviceinitialization(probe()function):
Disablethegenerationofnewinterrupts.
Ifyoudon't,thesystemwillgetspuriousinterrupts,andwill
eventuallydisabletheIRQline.Badforotherdevicesonthis
line!
ReleasetheIRQ
StopallDMAactivity.
NeededtobedoneafterIRQsaredisabled
(couldstartnewDMAs)
ReleaseDMAbuffers:streamingfirstandthenconsistent
ones.
30
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

PCIdeviceshutdown
Unregisterfromothersubsystems
UnmapI/Omemoryandportswithio_unmap().
Disablethedevicewithpci_disable_device().
UnregisterI/Omemoryandports.
Ifyoudon't,youwon'tbeabletoreloadthedriver.

31
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Usefulresources
Documentation/PCI/pci.txtinthekernelsources
AnexcellentguidetowritingPCIdrivers,whichhelped
ustowritetheseslides.
Book:EssentialLinuxdevicedrivers(PrenticeHall)
Averygoodandrecentbook!
http://freeelectrons.com/redirect/elddbook.html
Book:LinuxDeviceDrivers(O'Reilly)
Availableunderafreedocumentationlicense.
http://freeelectrons.com/community/kernel/ldd3/
Wikipedia:
http://en.wikipedia.org/wiki/Peripheral_Component_Interconnect

32
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Usefulresources(2)
PCIUtilities:http://mj.ucw.cz/pciutils.html
lspci:showsPCIbusanddeviceinformation
setpci:allowstomanipulatePCIdeviceconfiguration
registers
PCIvendoranddeviceidrepository:
http://pciids.ucw.cz/read/PC/

33
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Relateddocuments

Allourtechnicalpresentations
onhttp://freeelectrons.com/docs
Linuxkernel
Devicedrivers
Architecturespecifics
EmbeddedLinuxsystemdevelopment
FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Howtohelp
Youcanhelpustoimproveandmaintainthisdocument...
Bysendingcorrections,suggestions,contributionsand
translations
Byaskingyourorganizationtoorderdevelopment,consulting
andtrainingservicesperformedbytheauthorsofthese
documents(seehttp://freeelectrons.com/).
Bysharingthisdocumentwithyourfriends,colleagues
andwiththelocalFreeSoftwarecommunity.
Byaddinglinksonyourwebsitetoouronlinematerials,
toincreasetheirvisibilityinsearchengineresults.

FreeElectrons.Kernel,driversandembeddedLinuxdevelopment,consulting,trainingandsupport.http//freeelectrons.com

Linuxkernel
Linuxdevicedrivers
Boardsupportcode
Mainstreamingkernelcode
Kerneldebugging
EmbeddedLinuxTraining
Allmaterialsreleasedwithafreelicense!
UnixandGNU/Linuxbasics
Linuxkernelanddriversdevelopment
RealtimeLinux,uClinux
Developmentandprofilingtools
Lightweighttoolsforembeddedsystems
Rootfilesystemcreation
Audioandmultimedia
Systemoptimization

FreeElectrons
Ourservices
CustomDevelopment
Systemintegration
EmbeddedLinuxdemosandprototypes
Systemoptimization
Applicationandinterfacedevelopment
Consultingandtechnicalsupport
Helpindecisionmaking
Systemarchitecture
Systemdesignandperformancereview
Developmenttoolandapplicationsupport
Investigatingissuesandfixingtoolbugs

You might also like