2

I've been looking for the best StackExchange community to ask this. I hope it fits well here.

As a risk analyst in a trading business, I'm often confronted with large volumes of data, way beyond the excel-vba/MS Access scale capabilities.

However, when I requested my IT department to have a (any) SQL Database installed in my machine (or in whatever place) so I could 'play with', I was informed that it's an IT policy not to allow a mortal user like me to build a database because somehow I could threaten the infrastructure.

Assuming this reply is not due to my poor coding skills or my dubious character, can anyone put some perspective in this policy? Does anyone knows if this is a regular IT policy? Is there a way I could implement a SQL database that would eliminate such fears?

tks in advance!

3
  • Yes, unfortunately this is a regular IT policy. Most IT departments are only able to provide a standardized set of services, not the agile tool that computers can be. I think your best approach is to find a sponsor high up in the business with enough clout to get it done. Your IT department may or may not have a role to play in this. Commented Apr 23, 2014 at 14:50
  • 1
    From a DBA point of view: Regardless of how knowledgeable you are, this database will need to be managed at some point. If you leave the company, or if performance of your system slows down, it will take IT resources to deal with it. That being said, if IT isn't trying to actively solve your issue, they are forcing you to go around them to get your job done, which will create more work for them in the future. So while I understand their hesitation, the conversation can't stop at "NO", they need to work with you to find a solution that benefits the company. Commented Apr 23, 2014 at 15:02
  • @MarkWilkinson. You got the idea. With they current 'NO', all they get is me building some sort of db system in excel, which I'm sure is a worst solution for everybody, but honestly for me is quite an employment hedge. Reality is, I don't think they care.
    – VBOG
    Commented Apr 23, 2014 at 19:21

4 Answers 4

3

There are a variety of factors that would cause IT organizations to be cautious about creating databases and giving business users the level of access to those systems that you are, presumably, asking for when you talk about wanting to "play with" the data.

First off, since you're in a company that does trading, that implies that there are dozens of laws and agencies that regulate the company. In order to comply with those laws, your organization undoubtedly has put together dozens if not hundreds of policy documents that undoubtedly include policy statements that say things like only members of the DBA team have superuser privileges to modify data in databases, only changes that go through the official software development lifecycle are promoted to production, a separation of duties ensures that people that administer the database are different from people that code against the database which are different from people that input and maintain the data. The first step of the many, many audits that such an organization goes through is to check to see whether the policies in those documents matches the actual practice. Finding "rogue" databases that exist outside of these policies is a huge red flag in such an audit. Even if your database didn't actually create any real risk, it would create an audit finding because it violated the agreed-upon policies. Theoretically, the IT organization could work with their regulators to amend the dozens of policies to allow this sort of thing in special cases, but that would likely be a tremendous amount of work.

It is unlikely, though, that regulators would view the sort of risk analysis that you're talking about as the sort of thing that might get a pass from the burden of regulation. The process of building risk models and analyzing risk in a trading business is about as core a concern of regulators as you're going to find. Regulators are going to want someone to be able to explain exactly what sort of analysis is being done, what data sources are being used, how that data is being manipulated, etc. They're going to want to know who has access to that data, how it is backed up, etc.

Second, systems like the one you describe have a tendency to grow over time and often get dumped on IT organizations once they become unwieldy or when the original owners move on. A system that you set up today just for yourself to play with can quickly become a system that your department depends on and then a system that is integrated with all sorts of reporting and workflow processes as time goes on. At some point, though, you'll move on to another company or the system will get slower or someone will realize that some critical bit of the business depends on a system that is running with no backups on some guy's machine and then IT will get the call to own the system. Those projects are generally train wrecks. These systems aren't generally designed well (risk analysts generally come up with better software designs than software developers come up with risk models but both pursuits benefit from expert attention up front), they're not properly documented, etc. They also tend to rely on tons of manual processes that scale acceptably when one business user owns the process and understands in detail what the system is doing and why but that scales incredibly poorly when IT folks are trying to simultaneously manage many different systems that they don't understand in deep detail. IT organizations generally do everything they can to prevent these systems from sprouting in the first place because they turn into such hassles in the end.

Third, if the system that you're building is necessary for you to do your job, then it's likely that it needs the attention of IT. It probably needs to be backed up, for example. It probably needs to be added to various proactive and reactive monitoring scripts/ systems that the DBAs monitor in order to ensure that sufficient disk space is available, that the workload of your system isn't crushing other systems, etc. It probably needs to be part of the disaster recovery plan (if only to note that it's a low priority system to restore). It needs to get added to the set of systems that the DBAs patch when security updates come out or when the enterprise upgrades to new versions of the database. It needs to get added to the support contract so that it doesn't raise issues when the company gets audited by the vendor. These things are generally much harder when IT doesn't own the system. But if IT owns the system, it needs to behave like all the other IT systems-- it needs to be developed by developers, used by users, administered by administrators, it needs to have different environments (dev, test, staging, prod) to support the standard development models, etc.

In most IT organizations, if you need this sort of setup, you would need a project to create it (which would include things like assigning development resources, a project manager, etc.) That project obviously needs to be prioritized, implemented, etc.

If you really want "free reign" to do whatever you want without any of the controls that are put on developers, it's possible that you, your manager, and IT could get together and come up with a reasonable compromise. Perhaps you could be given slightly elevated privileges in some database/ schema somewhere along with restrictions in the types of things you're allowed to do (i.e. nothing from what you're doing can feed any other system, nothing can be used in a production process, etc.) It's not uncommon for analysts to have a "playground" where they can create tables to support ad hoc reporting, for example, with the understanding that anything that needs to become part of a production report needs to get built following the full development lifecycle process.

1
  • Tks. Indeed that's pretty much the perspective I get from my IT. I'm not arguing with the rule I was just wondering if there was a simple, safe and low cost way to implement a (limited) database outside (close) IT control. Wait, that's another question!.
    – VBOG
    Commented Apr 24, 2014 at 15:00
1

It's fairly common for IT departments to have these kinds of policies. Locked down systems are so much easier to manage, and most users have no real need for admin rights or a sandbox environment. There's a few common ways to get around it:

  1. Ask for a 2nd PC that's not connected to the network, so you can develop on it without threatening anything.

  2. Ask for VM software (such as Virtualbox) on your own PC or on a server. do your development in the VM.

  3. Bypass your IT department altogether and use a cloud provider for your dev environment. AWS can spin up a server in seconds for a few dollars. This might not be possible if the data is considered sensitive and/or your company has a policy against storing that data externally.

5
  • 4
    It might also be to comply with data management/protection policies and laws so be a bit careful about punting your data out to any cloud based service that hasn't been approved by your employers. Not that you can't do it, but you might be breaking your employer's rules which may well constitute gross negligence (breach of contract and therefore possibly a sacking offence). I have witnessed someone being sacked just for reading a co-workers data from a finance system for a joke. Commented Apr 23, 2014 at 14:48
  • Yeah good point, I will edit my answer appropriately Commented Apr 23, 2014 at 15:22
  • @AndrewBrennan. Two questions: a) Why not 1 with connection with the network? I just need other PCs in my department to be able to get data from the db. b) Any advantage of the VM machine approach to the 2nd PC approach from IT point of view?
    – VBOG
    Commented Apr 23, 2014 at 19:33
  • The idea of having a PC off the network is that you develop a "proof of concept" program on a machine that cannot interfere with anything else during the development process. i.e. when you break it, you won't take anything else down with it. When you have proved that it works and is stable and secure, then you can introduce it to the rest of the network. Commented Apr 24, 2014 at 10:26
  • Using a VM means you can only break the VM if you have admin rights and are doing software development. If it breaks, all you have to rebuild is the VM. There is still a risk of you using far too many resources and interfering with other apps with this approach, though. Commented Apr 24, 2014 at 10:28
0

As it's only on your machine you should investigate sqlite or another embedded/portable DB. I was able to bypass our locked down WinXP environment to run a web server and had scripted integration jobs from CSV and XML files.

SQLITE is incredibly fast and depending on the file system can easily cope with huge files and millions of records. I'm running 20gb of dbs now on win 7 and hosting pages and reports for the wider company.

2
  • Again this might be a breach of company policy and also would be much better off being hosted in a more official capacity Commented Apr 23, 2014 at 15:52
  • @Shand. Appreciate the tip but indeed I'm not sure I want to play the gray area game here. And to be honest I want this data as away from the web as possible.
    – VBOG
    Commented Apr 23, 2014 at 19:16
0

@VBOG There's not much of a difference with storing the data in excel or in a single sqlite DB when it comes to protecting the data.

IT departments seem to be quite comfortable with allowing users to build databases in spreadsheets, but as soon as a DB is mentioned they perk up and start interfering.

For 'play', sqlite will allow you to demonstrate a proof of concept and allow you to model your data properly.

As for the web, you don't need to share your site with anyone and it would remain securely on your machine.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.