OVAL Board Minutes

Teleconference - 2004-12-13, 13:00 - 14:15 EST (GMT -0500)

Attendees

Raffael Marty - ArcSight
Jay Beale - Bastille Linux
Chan Yoon - Bindview
Kent Landfield - Citadel
Dennis Moreau - Configuresoft
Mike Murray - nCircle
Pinkesh Shah - NetIQ
Jonathan Baker - MITRE
Andrew Buttner - MITRE
David Mann - MITRE
Robert Martin - MITRE
David Proulx - MITRE
Ingrid Skoog - MITRE
Matthew Wojcik - MITRE
Margie Zuk - MITRE
Dan Bezilla - Secure Elements

Agenda

• OVAL Status update
• Proposed name change:
  Open Vulnerability Assessment Language to Open Validation and Assessment Language
• SQL deprecation
• Schema acceptance process changes
• Version 4 schema review
• OVAL compatibility

Welcome to new and prospective Board members

• Dennis Moreau
• Mike Murray
• Pinkesh Shah
• Gerhard Eschelbeck [unable to attend]

OVAL Status update

• 993 definitions (up from ~850 in August)
• Version 4 taking most time
  • Cisco IOS schema draft posted
  • Debian Linux schema draft posted
• Compatibility efforts continuing
  • Better define compatibility - Requirements draft in progress
  • Processing compatibility declarations
• Better documentation of schema coming
• Definition writer state
  • Tool to facilitate creating OVAL Definitions
    • Avoid writing lots of XML by hand
    • Manage test library
    • Help co-ordinate efforts of multiple orgs writing definitions
    • For creating content for existing platform schemas, rather than to aid creating new schemas
  • Still in development
  • In use internally at MITRE for 2-3 months
  • Not ready for wide distribution--hope to be ready for limited release soon

Proposed name change:
Open Vulnerability Assessment Language to Open Validation and Assessment Language

OVAL was initially used for vulnerability assessment definitions, but intent has always been to support other applications of the language. OVAL lets you precisely define a machine state ("Vulnerable to CVE-2004-1234", "Not compliant with item 5 of my security policy", "Patch 8759 would be applicable") and then test real machines against that definition. Since policy compliance using OVAL is in development, it seems like a good idea to change the OVAL acronym expansion to reflect the broader scope.

Shah: Don't "validation" and "assessment" mean essentially the same thing?

Beale: We'd thought of validation in regards to testing machines for security policy compliance, as opposed to assessing them for vulnerabilities.

Wojcik: Maybe they aren't well-differentiated, but we don't want to change the OVAL name itself, and this is the best new expansion we've been able to come up with.

SQL deprecation

As was announced some time ago, SQL is to be deprecated as a format for OVAL. Continuing SQL support means double the work for MITRE in many areas and no longer makes sense. With the acceptance of Version 4, SQL will be completely deprecated.

Landfield: I think it's a great idea. Will support be dropped with Version 4, or are you looking to do this immediately?

Wojcik: Our plan was to wait until Version 4, but due to the way we've been working to convert to the new version, there have been some issues.

Baker: We've already converted the internal database which OVAL definitions are generated from to Version 4. Generating Version 3 content from that database is actually a bit difficult, and we've decided it's more development effort than we want to invest to generate SQL any more. So the SQL on the web site is now static, and not being updated.

Landfield: Should put something on the web site to make it more clear that SQL isn't being updated, so no one starts going down the SQL path now.

Wojcik: Good point, and we'll do that. As an aside, we're starting to get a better handle on web site statistics, and downloads of XML content have definitely passed SQL.

Shah: Do we have any insight into what OVAL users actually do when they download OVAL content and tools?

Wojcik: We don't have much feedback from users at this point, and we will think about how to get more. Since the OVAL Reference Interpreter is so far the only tool we're aware of that performs assessments with OVAL definitions, presumably people are using that. With the OVAL compatibility declarations already on the site and in the works, more tools will be becoming available.

Schema acceptance process changes

The original process for adopting a new schema version was modeled on the process for definitions. Changes were proposed and announced, and accepted in a fairly short time frame if there weren't further comments. With more applications being written to use OVAL, it's clear we need a more formal process with reasonable time allowed for development cycles. In previous discussions with the Board, the consensus seemed in favor of more significant changes less often, rather than frequent incremental revision.

The proposed new process is roughly:

• Planning / Discussion period
  • Feature requests, MITRE or otherwise
  • Open-ended time frame
    
• Draft / Internal development
  • Formal XSD schemas published on web site
  • No new feature requests accepted, but bug fixes allowed
  • MITRE updates all tools to new schema:
    • Back-end (database, definition generation scripts, web site)
    • Definition writer
    • Interpreter
    • Example files available on the web site
  • Intent is to find any issues with the draft
  • Other OVAL-compatible developers encouraged to work with the new
    version
    
• Release Candidate freeze
  • MITRE tools finished
  • No changes to schema unless critically needed
  • Developers of compatible expected to convert
  • Duration of this period is a question
    
• Acceptance
  • All content at oval.mitre.org available in new format
  • New schema version replaces old
  • Old format available for a time
    • Archived?
    • New content when possible?
      
  • End-of-life
    • Old format removed
    • When?

Landfield: The process seems quite reasonable. I like the idea that once a new version is accepted, no new content would be generated in the old format, and people are forced to move forward. As long as we've allowed the community reasonable time to convert, and we've made the time lines well known, it makes sense. We will have to do a better job on the web site to make clear what status the new proposed version is in, so everybody is aware and can project what they need to do.

Wojcik: Absolutely, and I'm first to admit the work on Version 4 hasn't followed this process very well, partly because we're still defining the process. From now on, we want to be much more clear.

One mistake we made this time was accepting new feature requests after we'd already started converting our tools, which meant re-coding various components a number of times. We need to avoid that in the future.

Shah: I agree with Kent, and think the process outlined makes sense. Having the earlier version available while the new version is being planned is key; no matter how much time is allowed for conversion, there will always be some development cycles that don't match. I think we should keep an old version available longer, so version 3 would be available until Version 5 is available. Once 5 is accepted, 3 might be not available even for download. I think we need a well-defined time frame, whether it's linked to the next version, or an absolute time frame.

Wojcik: Do you agree with Kent that we shouldn't release new content in (e.g.) 3 once 4 is accepted, but just have a static download available?

Shah: I do agree. I think we need to continue releasing new content in version 3 until version 4 is accepted, but once it's released, I think it's fair and provides an incentive to users and developers to use the latest and greatest. I don't think the effort of updating the old one should be borne by MITRE. If there are specific vendors who need to continue supporting the old format for whatever reason, they can take the burden of taking the new checks and if possible converting them to the old format for their customers.

Wojcik: A standing question is how long, or rather how short, can the Release Candidate period reasonably be? From our perspective at MITRE, we clearly want to move things along so the language is able to evolve and grow. We're not making changes gratuitously; we make changes because there have been demonstrated reasons to make them. If the acceptance cycle is too long, people wanting to use new features will be frustrated.

An example is the WMI test in the Version 4 draft. Board member David Waltermire of CIS requested it because there are policy-related functions he can't test except through WMI. If the Release Candidate phase in particular is too long, he'll be unable to proceed at some point.

So what is a reasonable time frame from the final freeze to when the web site switches over?

Marty: For me [reading the OVAL Results format], even two months would be more than enough.

[Wojcik editorial note: This raises the point that different categories of compatibility will require different levels of effort to convert to a new schema version.]

Landfield: The situation will get harder as we get further into the process and more people have written tools and have integrated OVAL into the core aspects of their product. It seems that setting any time frame will be a problem for somebody. We just have to establish what we think is fair to the community and fair to the innovation that we want to bring to the tools and the infrastructure. Three months is probably a reasonable time frame.

There might be an alternative that's not strictly time-based. It could be registration based, where developers come in and register that this is on the critical path for them, and they need to be able to convert. That would let MITRE go back to them and say that on such a date things will be converting.

Shah: Whether it's three months or six months--I'm leaning towards four--what are the kinds of things that need to happen in that time period? One is that vendors need to update their tools to use the new version or the new schema. Is there some kind of recertification process that also is needed, where maybe MITRE has to look at the tool and see that it is indeed working with the new version, or is that optional and left to the vendor?

Wojcik: That's a very good question. OVAL compatibility is still a work in progress, and we'll have to add that question to the list of considerations. Bob Martin, who's the CVE Compatibility lead, is taking the lead on OVAL Compatibility as well. Bob?

Martin: As we were talking, I realised that we're going to have to have requirements about how quickly tools move over, and that is a requirement. So people who get involve know that it's an ongoing effort, and it's not do it once and get the OVAL Compatibility label forever. But exactly how formal that will be, whether we'll need to get into retesting--we haven't even figured out what kind of testing we'll have initially. It's definitely an area we'll need to think about. If we establish a high level of testing required for one area, retesting when moving to a new version makes sense. But we haven't figured out how to do detailed testing yet, so for Version 4 this won't be a live issue yet.

Wojcik: We'll at the very least need to have a requirement in the declaration or compatibility questionnaire that lists what version you're compatible with.

Shah: I agree. I think we need something in the RC phase to at least do an incremental check, if not a thorough recertification, to make sure that these tools when they are released are really compatible with the new version.

So, if picking a time frame is difficult, I think it's based on a lot of these factors. If I were to pick something, it would be more on the upper side, between four to six months. I say that because of the time it's going to take to do an evaluation and recertification, both internally with the vendor's own QA process, and also with MITRE.

The other reason is that vendors release cycles often time every so many months. Just because we've created a new version of the OVAL schema, that might not be the sole driver for a company to release a new version. So even if the work gets done, a release may have to wait until other new features are added to the new application version.

So I think three months is a little on the aggressive side. Then again, vendors will have an opportunity to know about these things up front, so they won't have to wait until the freeze to get started. But for those reasons, I lean towards the upper side of the four to six months.

Marty: My only concern would be that we don't get important changes out fast enough. Version 4 is going to have really important changes, or really cool things, and if we have a six-month period, then we can't push new things fast enough.

Shah: I agree. It's a trade off we all have to think through: the time to market versus the time it takes to really engineer something.

Wojcik: We also don't want to penalize someone because they've just finished a major release a month before the new version is accepted.

This is really good feedback; we weren't looking to make a decision on a number of months today, but the input is very valuable.

Martin: Perhaps we could base it on, say, if 95% of the registered products have converted.

Wojcik: So there could be a maximum time limit, which might be long, and the opportunity to accept earlier if everybody's converted.

Another thing I was thinking about: maybe sometime say halfway through the freeze process, we do start making content available in the new version, even though it's not officially accepted yet. So people who've converted, their customers could start using it, with some kind of adviso.

Landfield: That was what I was getting at with the registration idea, where vendors could notify MITRE that the new version is in their critical path.

In the case of version 3, the only tool using it that I know of is Raffy and ArcSight. Once he's done, I don't know what would be holding us back on version 4.

Wojcik: I don't either. Maybe we should make the actual freeze period for version 4 shorter than we will in the future, because there isn't a lot of code that we know of that's actively using version 3. At MITRE, partly because of our learning process in handling this transition this time, we feel like the version 4 acceptance process has dragged on. There are things in version 4 that we think are important and will be very helpful for new development, and we don't want to see it drag on forever. Especially if it's only to satisfy an arbitrary time period.

Landfield: Is there anyone on here on the Board who would be negatively impacted by a swift move to version 4?

[Silence]

Landfield: I think silence begets acceptance of a shorter term this period.

[Agreement]

Version 4 schema review

Status of version 4:

• New acceptance policy not quite followed so far
• No new features for version 4!
• Almost done internal development phase
  • Interpreter conversion still in progress
  • Back end database, scripts, definition writer all converted
• MITRE would like to enter Release Candidate / freeze on 3 Jan 2005
• Examples of Definitions, System Characteristics, and OVAL Results available by 17 December 2004 (Friday)
• Review needed! Any bug fix changes must be in before RC phase.

Major changes in version 4:

• 2 new platforms (Cisco IOS, Debian Linux)
• Object specifier separated from tested data in tests
• System Characteristics not tightly coupled to specific tests
• SC object instance IDs--for example, if a regular expression in a file path which matched 5 files, each file matched has an instance ID
• OVAL Results file can link to SC file using instance ID
• Documentation is being included in the XSD for the schemas, so there are structural changes to the schema files themselves which don't affect validation

See Proposed Schema page:
http://oval.mitre.org/language/schemachanges.html

OVAL compatibility

A requirements document for compatibility is in progress and will be sent to the Board for comment when ready. Anyone who hasn't reviewed the compatibility information currently available on the website should do so. Comments or requests on compatibility are always welcome either to MITRE directly or on the Board list.

Web site comment

Landfield: It seems that email archives on the web site of the various lists aren't updated very frequently--sometimes every six weeks instead of every mail message.

Wojcik: Yes, that's been a problem. We're working to improve that in the very near future, although it's likely to be a daily update rather than an instant one.

We're also working on a broader web site redesign, to better organize the site and improve usability. Any comment is welcome.

System inventory using OVAL

Raffael Marty has raised the issue that it would be useful to be able to get basic system inventory information with OVAL. In addition to facts like "Vulnerable to CVE-2004-1234", "Not compliant with item 5 of my security policy", and "Patch 8759 would be applicable", applications using OVAL might well want to report operating system installed, application versions, what services are listening to the network, etc.

Clearly OVAL is well positioned to determine these facts. However, the current definition classes don't match this application well. Information like OS type and version is available (obviously, since it's tested in the great majority of definitions!), but not easily accessible with a well-known name. It may even be spread across the System Characteristics and OVAL Results files.

The question is, is this a use case OVAL should support? It would certainly be possible to write OVAL definitions to perform these tasks, but should OVAL add a software or system inventory class of definitions to allow standardization?

Moreau: Because that sort of information is pivotal to many remediation decisions, whether it's remediation or patch management or settings changes, it's a real logical extension to the language. So we should be sure we can support it.

Wojcik: One issue we've thought of is there could be an explosion in the number of OVAL definitions, if we make this new class of system inventory defs. So does it make sense to take that approach, or is there a more elegant way of handling it.

Another consideration is that some system inventory definitions would boil down to essentially a single test, and a test which is currently used by many current vulnerability definitions. For example the "Windows 2000 is installed" test. So would it make sense to let OVAL definitions include the results of other definitions.

I do think this is a natural application for OVAL, and a use case that anybody who's using OVAL in their product is going to get into. So it seems we should support it.

I do want to be clear that this is forward-looking at this point, though, and we're not suggesting any changes to Version 4 for this! We wanted to put it out there for people to start thinking about.

Moreau: Just to give you empirical validation of this idea, we have this kind of information in our product now, and our largest customer site use the confluence of non-compliance and system inventory all the time. So it seems to be compelling at least in the large scale enterprises

Page Last Updated: February 07, 2008