 |
History
| Log In
|
|
|
|
[
Permlink
| « Hide
]
Jelmer Kuperus [29/Jun/05 02:24 PM]
patches are for cc 2.2.1
I like the idea. A few comments:
- "dir" is the traditional attribute name for a directory to look for things in; this might be a better choice than "location". - would it be a good idea to support a nested <fileset/>? That would add flexibility, and make it consistent with the way (most) of the rest of CC/Ant/etc specify this sort of information. The idea of including a "check for mods" interval is good. The attribute was called location because it can also point to a single file.
I think the nested fileset would be very powerfull, it seems like a good idea, I'll look into that. Hi.
That's very nice to have this in place. In fact I need that :) I second the request of using filesets to define sources. But I am not sure this is the right solution (at least when it comes to my needs). It's OK for reading, but not for notitications. Currently CC looks at the config in a monolithic way. For example, config change detection can only happen at a global level. I would like it to look at it in a more granular way and be able to detect single project changes. Hi Jerome,
I am not sure if I fully understand what you are saying about notification. At some point you would probably want to have the ability to reload the entire monolithic config.xml. I looked into that but that seemed very dicey and involved The patches I made are less drastic and still manage to meet our needs I'll describe briefly how I set things up here. We have one config file per project located in a folder projects-available. We have another folder called projects-enabled that contains symlinks to the files in projects-available The config.xml we use just contains 3 lines <cruisecontrol> <include location="projects-available" interval="10" /> </cruisecontrol> Behind the scenes A FileWatcher checks for any create / update / deletes in this folder. and calls the respective callbacks on the CruiseControlController that take care of starting / stopping / updating any affected projects. All without affecting other projects. Adding a project just becomes dropping an xml file in a folder, and making a symlink to it. In at most 10 seconds it will be up and running. If it doesn't run you can check the console output / log see whats wrong and update or touch the config file for it to try again If you want to (temporarily) disable a project you just remove the symlink in projects-available. you don't loose the actual file. which is much easier than commenting out sections in config.xml Updates will be automatically detected. > I am not sure if I fully understand what you are saying about notification.
I want that changes to the config result in timely build loop updates and UI reflecting the current state of the config. > At some point you would probably want to have the ability to reload the entire monolithic config.xml. > I looked into that but that seemed very dicey and involved See > The patches I made are less drastic and still manage to meet our needs > > I'll describe briefly how I set things up here. > > We have one config file per project located in a folder projects-available. > We have another folder called projects-enabled that contains symlinks to the files in projects-available You guys follow setup used by Debian (and perhaps other) to configure apache2, right? :) > The config.xml we use just contains 3 lines > > <cruisecontrol> > <include location="projects-available" interval="10" /> > </cruisecontrol> > > Behind the scenes A FileWatcher checks for any create / update / deletes in this folder. and calls the respective callbacks on the > CruiseControlController that take care of starting / stopping / updating any affected projects. All without affecting other projects. That's great. > Adding a project just becomes dropping an xml file in a folder, and making a symlink to it. In at most 10 seconds it will be up and running. > > If it doesn't run you can check the console output / log see whats wrong and update or touch the config file for it to try again > > If you want to (temporarily) disable a project you just remove the symlink in projects-available. > you don't loose the actual file. which is much easier than commenting out sections in config.xml > > Updates will be automatically detected. I will be interested by something like that. Could you make your code available? J Ok i've updated the patch to reflect the suggestions made.
The location attribute was removed and instead 4 new attributes where introduced dir : the root of the directory tree of this FileSet. (either dir or file is required) file : shortcut for a single file fileset. (either dir or file is required) includes : comma- or space-separated list of patterns of files that must be included; all files are included when omitted. (optional) excludes : comma- or space-separated list of patterns of files that must be excluded; no files (except default excludes) are excluded when omitted. (optional) includes and excludes accept ant style patterns Some example configs : Config 1. Tries to include all xml files the projects dir and subdirectories Checks for modifications every 20 seconds <cruisecontrol> <include dir="projects" includes="**/*.xml" interval="20" /> </cruisecontrol> Config 2. Tries to include all xml files the projects dir but not from its subdirectories Does not check for modifications <cruisecontrol> <include dir="projects" includes="*.xml" /> </cruisecontrol> Config 3. Tries to include projects/project1.xml Checks for modifications every 20 seconds <cruisecontrol> <include file="projects/project1.xml" interval="20" /> </cruisecontrol> Config 4. Tries to include all files in the projects dir and subdirectories but excludes any *.txt files <cruisecontrol> <include dir="projects" includes="**/*" excludes="**/*.txt" interval="20" /> </cruisecontrol> Jerome,
> You guys follow setup used by Debian (and perhaps other) to configure apache2, right? :) yep, I allways found that a breeze to manage >I will be interested by something like that. Could you make your code available? I attached a patch, and just updated a new one that reflects suggested changes. It would be great if something like this could be added to the main cc branch Should the interval be an attribute of the include? Or should it be a general CC build loop config?
Maybe the refresh rate should be global, at least in the first implementation. Maybe defined as a system attribute. I am thinking of that in combination with the mecanism proposed in I think this feature is great. I just would like it to fit with the rest. I haven't fully reviewed the
I think it would just be a matter of implementing a ConfigReloader that triggers a config file reload on callbacks made by FileWatchers I kind of like the idea of having fine grained control over the interval on a per dir or per file basis however setting it globally might be more conventient in a lot of cases. Maybe we could have both ? where the specied interval overrides the global interval I think the simplest thing that could possibly work is a non recursive design...
<cruisecontrol> <include dir="projects" includes="fred/cruisecontrolconfig.xml" /> <include dir="projects" includes="wilma/cruisecontrolconfig.xml" /> <include dir="projects" includes="rubble/barney/cruisecontrolconfig.xml" /> <include dir="projects" includes="rubble/bambam/cruisecontrolconfig.xml" /> </cruisecontrol> ... with an unspecified refresh interval. Improvements can be worked out later... Hi Jelmer,
Have you tested this in combination with properties support? Can you use global properties and set and use local properties within an included file? Cheers, Robert Paul,
I agree as long as you could still specify non recursive wildcards (eg. *.xml) such a solution would work for us. However are you suggesting dropping the interval attribute and using a global setting or dropping change detection all together? Because having cruisecontrol detect new and updated configs is vital to us. Robert, I assume you are referring to the ant style properties support as described at http://confluence.public.thoughtworks.org/display/CC/ConfigFiles If that is the case then no, not yet, the include file patches are for cc 2.2.1. However I'd be happy bring it up to date for the latest cvs version and make sure they are supported Anonymous, It seems to do a lot of the same, I am not sure if I would like having ant as a runtime dependency though I agree, the idea of having the config file in multiple pieces is related to
note that the 2.2.2 release candidate will reload the config file and handle added/removed project. to what extent does that remove the need for this patch?
The splitting of the config into various files is interesting for various reasons. In particular use file system access rights to allow users to modify particular parts of the config.
But now that the automatic reload happens, nothing prevents the splitting to be done anyway and an external process to recombine the bits and pieces into a single file when modifications to the parts are detected. The only drawback to this would be that this operation would have to be done differently on each platform? I don't think it warrants modifying the config file structure for that. In fact it could probably be done without modifying the structure: while still keeping the -configfile build loop argument, let's define a -configfiles "filesetpattern" argument. This should solve most cases without having to change the config file structure. In your case you would probably use something like projects/**/cruisecontrolconfig.xml Jelmer, would something like that satisfy you? Do you really need to specify each and every single project file? The issue that splits the config file into several parts.
One drawback of my proposed solution is the handling of the global properties and system definitions.
I still believe that one should split the responsibility of reading the config file (build loop) with the responsibility of assembling it. Whether it should be the role of a sub-process of the build loop to assemble the parts, it shouldn't force the config file to be modified. The -configfile suggestion seems fine, that would fullfill our needs
>But now that the automatic reload happens, nothing prevents the splitting to be done anyway and an external process to recombine the >bits and pieces into a single file when modifications to the parts are detected. The only drawback to this would be that this operation >would have to be done differently on each platform? Can you elaborate a bit on this. It sounds like you are proposing a shell script to do the recombining since it would be different on each platform. How would cc detect the changes ? , why wouldn't cruisecontrol do the recombining? Jelmer, [re 14/Jul/05 01:45 AM]
I'm suggesting just implementing a default design for the first version. Say an interval that is hard coded at 5 mins. That a file system refresh, nothing to do with SCM. That SCM under normal project build intervals updates, and that may have an affect on the compositie form of config.xml is an intereting side effect. And yes, I'm also, for the initial version, suggesting that a non recursive design is easiest to complete, commit, and get into a release. The longer spent debating the fine grained design, the longer it will be before anything is implemented. Snippets of config.xml being under SCM for the projects that they run against will be a killer feature of CC when shipped :-) - Paul Oh, and the interim solution of entity includes is just too brittle. One malformed include corrupts the entire config. Aside from the fact that it is not universally supported by xml parsers.
I will try to work on the design idea I gave in the forthcoming days. I want to have this setup working on my moca server.
remind me again, what's wrong with using XML includes?
we already support adding/removing of projects when the config file changes and if you're using xml includes it will detect if one of the included files changes. why is important to reimplement this feature which is already part of xml? The issue is that xml-include is brittle. It may work today but one misplaced < in an included file and all projects will cease to be built.
This is likely. The aspect of this that's really great is that you can suggest to individual teams that they maintain their own CC config, making fine graned changes to it and releasing the CC adminr from the chore of having to maintain it. Think 100 projects in the same SCM, all in their own directories. with a custom xml inclide frature hinged around <project/> then it one is busted, then it will simply be marked as such in the results page, as all other build fine. - Paul Another thing against using xml-includes here IMO is that it was very kind to just place some xml-files in a directory that then are validated and loaded.
(And that without touching the config.xml) What I would like to see is that I can do the following: 1) Specifiying a config-file that contains my projects (current implementation) 2) Specify a directory => all valid-cc-config-files are loaded with default-options around (single thread, no pre-configured plugins) 3) Specify both => the config-file is loaded and all files from directory, too Michael I really like Paul's point about being able to manage many projects independently.
I wonder if we could get the sort of behavior he describes using XInclude: http://www.w3.org/TR/2004/PR-xinclude-20040930/ unlike xml entities XInclude has a "fallback" element so that one misplaced < doesn't bring everything down. from the spec: "Failure to load an external entity is normally a fatal error. XInclude allows the author to provide default content that will be used if the remote resource cannot be loaded." Activating XInclude seems to be at most a three line change to the XML parsing code:
http://xerces.apache.org/xerces2-j/faq-xinclude.html I gather this feature did not get rolled into subsequent releases. I am using CC 2.4.0 and this functionality is just what we need! Is it possible to get a patch for 2.4.0?
This is still my most wished for CC feature.
To reiterate .... XML's Xinclude will not do it as one project's broken or missing XML will cause the XML file to be broken and thus Cruise to choke and die. And can it be commit in two stages, to please everyone ? First support for - <cruisecontrol> <include dir="projects" includes="fred/cruisecontrolconfig.xml" /> <include dir="projects" includes="wilma/cruisecontrolconfig.xml" /> <include dir="projects" includes="rubble/barney/cruisecontrolconfig.xml" /> <include dir="projects" includes="rubble/bambam/cruisecontrolconfig.xml" /> </cruisecontrol> Then support for - <cruisecontrol> <include dir="projects" includes="*.xml" /> </cruisecontrol> I have committed an initial implementation with the syntax:
<cruisecontrol> <include.projects file="relative/path/config.xml"/> </cruisecontrol> Plans are to add dir, include and exclude attributes. Also need to solve the reloading issue. (I might adopt the timer based approach over the current one triggered by the build queue.) Hi,
I'm using the project include stuff ( delivered in cruisecontrol-src-2.6 ): <cruisecontrol> <include.projects file="relative/path/config.xml"/> </cruisecontrol> There are few issues with this approach - unlike having projects inside main config.xml, projects from includes xml files doesn't re-read their config file when it changes, unless main config file is changed. Moreover, if I remove the <include.projects> tag from main config xml file, the project is not removed from buildqueue unless cruisecontroll is restarted. I'm not sure if this is the right behavior, and will try to find some solution (and post a patch) Regards, Jindrich Is not removing the project from the buildqueue really a problem? If I recall correctly when it comes time for the project to build it will find that it is stopped and just return right away.
The issue of not refreshing when the included file is changed is work that I just didn't get to yet (that's the reloading issue of the comment prior to your) but I hope to address it in 2.6.2. I've commited a fix such that changes to the included xml files are detected and the configuration reloaded appropriately. That fix will be part of CC 2.6.2.
Thank you Jeffrey, that's made a really useful feature even better! :-)
Hi,
I'm trying to use the tag include.projects to nest project files ( cruisecontrol-src-2.6.2): config.xml: <cruisecontrol> <include.projects file="config-workset.xml"/> </cruisecontrol> config-workset.xml: <cruisecontrol> <include.projects file="config-project1.xml"/> <include.projects file="config-project2.xml"/> </cruisecontrol> But the included files in the included file are not processed. Is nesting supposed to work with the tag incluce.projects? Greetings Bastian it was supposed to but there is a bug with the following one line fix:
Index: src/net/sourceforge/cruisecontrol/CruiseControlConfig.java =================================================================== --- src/net/sourceforge/cruisecontrol/CruiseControlConfig.java (revision 3072) +++ src/net/sourceforge/cruisecontrol/CruiseControlConfig.java (working copy) @@ -121,6 +121,8 @@ & | |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Improvement
Open
Major
[PATCH] include element
