LArSoft  v09_90_00
Liquid Argon Software toolkit - https://larsoft.org/
MixHelper.h
Go to the documentation of this file.
1 #ifndef art_Framework_IO_ProductMix_MixHelper_h
2 #define art_Framework_IO_ProductMix_MixHelper_h
3 // MixHelper
5 //
6 // Class providing, at construction time, registration services to
7 // users' "detail" mixing classes. (A "detail" class is the template
8 // argument to the instantiation of the "MixFilter" module template.)
9 //
11 // Configuration.
12 //
13 // MixHelper will be passed the configuration of the module. The
14 // following items are significant:
15 //
16 // fileNames (default empty)
17 //
18 // Sequence of secondary files for mixing. However, see the function
19 // registerSecondaryFileNameProvider(...) below. If a secondary file
20 // name provider is *not* registered, it is an error to have an
21 // empty fileNames.
22 //
23 // readMode (default sequential).
24 //
25 // Specify how events should be chosen from each file. Valid values
26 // are:
27 //
28 // sequential -- read the secondary events in order
29 // randomReplace -- random with replacement
30 // randomLimReplace -- events unique within a primary event
31 // randomNoReplace -- events guaranteed to be used once only.
32 //
33 // MT note: What gets mixed is unpredictable when multiple mixing
34 // modules run in parallel either because of multiple trigger
35 // paths or because of streams when the mixing mode is
36 // Mode::SEQUENTIAL. For art 3.0, mixing will be entirely
37 // serialized.
38 //
39 // coverageFraction (default 1.0).
40 //
41 // Ratio of sampled to total events in a file. Used by randomReplace
42 // and randomLimReplace modes only.
43 //
44 // wrapFiles (default false).
45 //
46 // Re-start from fileNames[0] after secondary events are
47 // exhausted. If this is false, exhausting the secondary file stream
48 // will result in the filter throwing an exception that is handled
49 // based on the configured handling policy for FileReadError
50 // exceptions.
51 //
52 // compactMissingProducts (default false).
53 //
54 // In the case of a secondary event missing a particular product,
55 // the sequence of product pointers passed to the MixOp will be
56 // compacted to remove nullptrs.
57 //
59 // readMode()
60 //
61 // Return the enumerated value representing the event mixing strategy.
62 //
64 // registerSecondaryFileNameProvider(<function> func)
65 //
66 // Register the provided function as a provider of file names for
67 // mixing. This should be called from the constructor of your detail
68 // object.
69 //
70 // <function> must be convertible to std::function<std::string()>. A
71 // free function taking no arguments and returning std::string, a
72 // functor whose operator() has the same signature, or a bound free or
73 // member function whose signature after binding is std::string() are
74 // all convertible to std::function<std::string()>.
75 //
76 // E.g. for a detail class with member function std::string getMixFile():
77 //
78 // registerSecondaryFileNameProvider(std::bind(&Detail::getMixFile, this));
79 //
80 // Notes:
81 //
82 // 1. It is a configuration error to provide a non-empty fileNames
83 // parameter to a module which registers a file name provider.
84 //
85 // 2. If the file name provider returns a string which is empty, the
86 // MixFilter shall thenceforth return false.
87 //
88 // 3. If the file name provider returns a non-empty string that does
89 // not correspond to a readable file, an exception shall be thrown.
90 //
92 // declareMixOp templates.
93 //
94 // These function templates should be used by writers of
95 // product-mixing "detail" classes to declare each product mix
96 // operation.
97 //
98 // All of the declareMixOp(...) function templates have the following
99 // template arguments:
100 //
101 // 1. The BranchType (defaults to art::InEvent). Specify explicitly if
102 // you wish to mix subrun or run products, or if you need to
103 // specify explicitly any of the other template arguments.
104 //
105 // 2. The incoming product type (deduced from the provided callable
106 // mixer argument).
107 //
108 // 3. The outgoing product type (deduced from the provided callable
109 // mixer argument).
110 //
111 // A product mixing operation should be specified by providing:
112 //
113 // 1. an InputTag specifying which secondary products should be mixed;
114 //
115 // 2. an optional instance label for the mixed product (defaulting to
116 // the instance label of the incoming product if unspecified); and
117 //
118 // 3. a callable mixer such as:
119 //
120 // bool mixfunc(std::vector<PROD const*> const&,
121 // OPROD&,
122 // PtrRemapper const&),
123 //
124 // As the user may prefer, the mixer may take the form of:
125 // a) an arbitrarily-named free function, or
126 // b) a function object whose operator() must have a signature
127 // whose arguments match those above, or
128 // c) an arbitrarily-named member function of any class.
129 // In each case, the mixer must have the same type (i.e.,
130 // the same return type and the same parameter types) illustrated
131 // by "mixfunc" above. The return value of the mix function is
132 // taken to indicate whether the product should be placed in the
133 // event.
134 //
135 // Generally speaking the PROD and OPROD template arguments are
136 // deducible from the callable mixer function which, once bound
137 // (if appropriate) to any provided class instance, should be
138 // convertible to an instantiation of the template instance
139 // art::MixFunc<PROD, OPROD>. See
140 // art/Framework/IO/ProductMix/MixTypes.h for more details.
141 //
142 // If the provided function is a member function it may be
143 // provided bound to the object upon which it is to be called by
144 // the user (in which case it is treated as a free function by the
145 // registration method) or by specifying the member function
146 // followed by the object to which it should be bound (in which
147 // case the bind will be done for the user). In this latter case
148 // the template argument specifying the product type need *not* be
149 // specified usually as it may be deduced from the signature of
150 // the provided function. If one specifies an overload set however
151 // (e.g. in the case where a class has several mix() member
152 // functions, each one with a different mix function) then the
153 // template argument must be specified in order to constrain the
154 // overload set to a single function.
155 //
156 // See also the description of the template alias
157 // art::MixFunc<PROD, OPROD> defined in
158 // art/Framework/IO/ProductMix/MixTypes.h.
159 //
160 // 4. An optional boolean, "outputProduct," defaulting to, "true." A
161 // false value for this parameter indicates that the mix product
162 // will *never* be put into the event and should therefore not be
163 // declared. If the mix operation so declared ever returns true an
164 // exception will be thrown.
165 //
166 // declareMixOp() may be called with any of the following argument
167 // combinations:
168 //
169 // 1. Provide an InputTag and a mixer that is a free function or
170 // function object (wrapped as a MixFunc<PROD, OPROD>).
171 //
172 // 2. Provide an InputTag, an output instance label, and a mixer
173 // that is a free function or function object.
174 //
175 // 3. Provide an InputTag, a mixer that is a non-const member
176 // function (of any class), and an object to which that member
177 // function should be bound.
178 //
179 // 4. Provide an InputTag, an output instance label, a mixer that is
180 // a non-const member function (of any class), and an object to
181 // which that member function should be bound.
182 //
183 // 5. Same as 3, but providing a mixer that is a const member function.
184 //
185 // 6. Same as 4, but providing a mixer that is a const member function.
186 //
187 // Note: For signatures 3-6, if the compiler complains about an
188 // unresolved overload your first move should be to specify the
189 // product type(s) as template argument (to do this you must specify
190 // explicitly the BranchType template argument first). If that does
191 // not resolve the problem, try an explicit:
192 //
193 // const_cast<T const&>(t)
194 //
195 // or
196 //
197 // const_cast<T&>(t)
198 //
199 // as appropriate.
200 //
202 // produces() templates.
203 //
204 // Call as you would from the constructor of a module to declare
205 // (e.g., bookkeeping) products to be put into the event that are
206 // *not* direct results of a single product mix operation. For the
207 // latter case, see the declareMixOp() templates above.
208 //
209 // Signatures for produces():
210 //
211 // 1. produces<PROD>(optional_instance_name);
212 //
213 // Register a product to go into the event.
214 //
215 // 2. produces<PROD, art::InRun>(optional_instance_name);
216 // produces<PROD, art::InSubRun>(optional_instance_name);
217 //
218 // Register a product to go into the run or subrun.
219 //
221 
222 #include "CLHEP/Random/RandFlat.h"
232 #include "cetlib/exempt_ptr.h"
233 #include "fhiclcpp/fwd.h"
234 #include "fhiclcpp/types/Atom.h"
235 #include "fhiclcpp/types/Sequence.h"
236 
237 #include <functional>
238 #include <iosfwd>
239 #include <memory>
240 #include <string>
241 #include <vector>
242 
243 namespace art {
244 
246  using ProviderFunc_ = std::function<std::string()>;
247 
248  public:
249  enum class Mode {
250  SEQUENTIAL = 0,
254  UNKNOWN
255  };
256 
257  struct Config {
258  fhicl::Sequence<std::string> filenames{fhicl::Name{"fileNames"}, {}};
259  fhicl::Atom<bool> compactMissingProducts{
260  fhicl::Name{"compactMissingProducts"},
261  false};
262  fhicl::Atom<std::string> readMode{fhicl::Name{"readMode"}, "sequential"};
263  fhicl::Atom<double> coverageFraction{fhicl::Name{"coverageFraction"},
264  1.0};
265  fhicl::Atom<bool> wrapFiles{fhicl::Name{"wrapFiles"}, false};
267  };
268 
269  explicit MixHelper(Config const& config,
270  std::string const& moduleLabel,
271  ProducesCollector& collector,
272  std::unique_ptr<MixIOPolicy> ioHandle);
273  explicit MixHelper(fhicl::ParameterSet const& pset,
274  std::string const& moduleLabel,
275  ProducesCollector& collector,
276  std::unique_ptr<MixIOPolicy> ioHandle);
277  ~MixHelper();
278 
279  // Returns the current mixing mode.
280  Mode readMode() const;
281 
282  // Registers a callback to the detail object to determine the next
283  // secondary file to read.
285 
286  // A.
287  template <class P>
288  void produces(std::string const& instanceName = {});
289 
290  // B.
291  template <class P, BranchType B>
292  void produces(std::string const& instanceName = {});
293 
294  // 1.
295  template <art::BranchType B = art::InEvent, typename PROD, typename OPROD>
296  void declareMixOp(InputTag const& inputTag,
297  MixFunc<PROD, OPROD> mixFunc,
298  bool outputProduct = true);
299 
300  // 2.
301  template <art::BranchType B = art::InEvent, typename PROD, typename OPROD>
302  void declareMixOp(InputTag const& inputTag,
303  std::string const& outputInstanceLabel,
304  MixFunc<PROD, OPROD> mixFunc,
305  bool outputProduct = true);
306 
307  // 3.
308  template <art::BranchType B = art::InEvent,
309  typename PROD,
310  typename OPROD,
311  typename T>
312  void declareMixOp(InputTag const& inputTag,
313  bool (T::*mixfunc)(std::vector<PROD const*> const&,
314  OPROD&,
315  PtrRemapper const&),
316  T& t,
317  bool outputProduct = true);
318 
319  // 4.
320  template <art::BranchType B = art::InEvent,
321  typename PROD,
322  typename OPROD,
323  typename T>
324  void declareMixOp(InputTag const& inputTag,
325  std::string const& outputInstanceLabel,
326  bool (T::*mixfunc)(std::vector<PROD const*> const&,
327  OPROD&,
328  PtrRemapper const&),
329  T& t,
330  bool outputProduct = true);
331 
332  // 5.
333  template <art::BranchType B = art::InEvent,
334  typename PROD,
335  typename OPROD,
336  typename T>
337  void declareMixOp(InputTag const& inputTag,
338  bool (T::*mixfunc)(std::vector<PROD const*> const&,
339  OPROD&,
340  PtrRemapper const&) const,
341  T const& t,
342  bool outputProduct = true);
343 
344  // 6.
345  template <art::BranchType B = art::InEvent,
346  typename PROD,
347  typename OPROD,
348  typename T>
349  void declareMixOp(InputTag const& inputTag,
350  std::string const& outputInstanceLabel,
351  bool (T::*mixfunc)(std::vector<PROD const*> const&,
352  OPROD&,
353  PtrRemapper const&) const,
354  T const& t,
355  bool outputProduct = true);
356 
357  // Random number engine creation
360  std::string const& kind_of_engine_to_make);
362  std::string const& kind_of_engine_to_make,
363  label_t const& engine_label);
364 
366  // Mix module writers should not need anything below this point.
368  bool generateEventSequence(size_t nSecondaries,
369  EntryNumberSequence& enSeq,
370  EventIDSequence& eIDseq);
372  EntryNumberSequence const&);
373  void mixAndPut(EntryNumberSequence const& enSeq,
374  EventIDSequence const& eIDseq,
375  Event& e);
376  void setEventsToSkipFunction(std::function<size_t()> eventsToSkip);
377 
378  private:
379  MixHelper(MixHelper const&) = delete;
380  MixHelper& operator=(MixHelper const&) = delete;
381 
382  using MixOpList = std::vector<std::unique_ptr<MixOpBase>>;
383 
384  cet::exempt_ptr<base_engine_t> initEngine_(seed_t seed, Mode readMode);
385  std::unique_ptr<CLHEP::RandFlat> initDist_(
386  cet::exempt_ptr<base_engine_t> engine) const;
387  bool consistentRequest_(std::string const& kind_of_engine_to_make,
388  label_t const& engine_label) const;
389  Mode initReadMode_(std::string const& mode) const;
390  bool openNextFile_();
391 
393  MixOpList& mixOps);
394 
396  std::string const moduleLabel_;
397  std::vector<std::string> const filenames_;
404  double const coverageFraction_;
405  std::size_t nEventsReadThisFile_{};
406  std::size_t totalEventsRead_{};
407  bool const canWrapFiles_;
410  cet::exempt_ptr<base_engine_t> engine_;
411  std::unique_ptr<CLHEP::RandFlat> dist_;
412  std::function<size_t()> eventsToSkip_{};
413  EntryNumberSequence shuffledSequence_{}; // RANDOM_NO_REPLACE only.
414  bool haveSubRunMixOps_{false};
415  bool haveRunMixOps_{false};
417 
418  std::unique_ptr<MixIOPolicy> ioHandle_{nullptr};
419  };
420 
421  std::ostream& operator<<(std::ostream&, MixHelper::Mode);
422 }
423 
424 inline auto
426 {
427  return readMode_;
428 }
429 
430 // A.
431 template <class P>
432 inline void
433 art::MixHelper::produces(std::string const& instanceName)
434 {
435  collector_.produces<P>(instanceName);
436 }
437 
438 // B.
439 template <class P, art::BranchType B>
440 inline void
441 art::MixHelper::produces(std::string const& instanceName)
442 {
443  collector_.produces<P, B>(instanceName);
444 }
445 
446 // 1.
447 template <art::BranchType B, typename PROD, typename OPROD>
448 inline void
450  MixFunc<PROD, OPROD> mixFunc,
451  bool outputProduct)
452 {
453  declareMixOp<B>(inputTag, inputTag.instance(), mixFunc, outputProduct); // 2.
454 }
455 
456 // 2.
457 template <art::BranchType B, typename PROD, typename OPROD>
458 void
460  std::string const& outputInstanceLabel,
461  MixFunc<PROD, OPROD> mixFunc,
462  bool outputProduct)
463 {
464  if (outputProduct) {
465  produces<OPROD>(outputInstanceLabel);
466  }
467  if (B == art::InSubRun) {
468  haveSubRunMixOps_ = true;
469  } else if (B == art::InRun) {
470  haveRunMixOps_ = true;
471  }
472  mixOps_.emplace_back(new MixOp<PROD, OPROD>(moduleLabel_,
473  inputTag,
474  outputInstanceLabel,
475  mixFunc,
476  outputProduct,
478  B));
479 }
480 
481 // 3.
482 template <art::BranchType B, typename PROD, typename OPROD, typename T>
483 inline void
485  bool (T::*mixFunc)(std::vector<PROD const*> const&,
486  OPROD&,
487  PtrRemapper const&),
488  T& t,
489  bool outputProduct)
490 {
491  declareMixOp<B>(inputTag,
492  inputTag.instance(),
493  mixFunc,
494  t,
495  outputProduct); // 4.
496 }
497 
498 // 4.
499 template <art::BranchType B, typename PROD, typename OPROD, typename T>
500 void
502  std::string const& outputInstanceLabel,
503  bool (T::*mixFunc)(std::vector<PROD const*> const&,
504  OPROD&,
505  PtrRemapper const&),
506  T& t,
507  bool outputProduct)
508 {
509  using std::placeholders::_1;
510  using std::placeholders::_2;
511  using std::placeholders::_3;
512  declareMixOp<B>(
513  inputTag,
514  outputInstanceLabel,
515  static_cast<MixFunc<PROD, OPROD>>(std::bind(mixFunc, &t, _1, _2, _3)),
516  outputProduct); // 2.
517 }
518 
519 // 5.
520 template <art::BranchType B, typename PROD, typename OPROD, typename T>
521 inline void
523  bool (T::*mixFunc)(std::vector<PROD const*> const&,
524  OPROD&,
525  PtrRemapper const&) const,
526  T const& t,
527  bool outputProduct)
528 {
529  declareMixOp<B>(inputTag,
530  inputTag.instance(),
531  mixFunc,
532  t,
533  outputProduct); // 6.
534 }
535 
536 // 6.
537 template <art::BranchType B, typename PROD, typename OPROD, typename T>
538 void
540  std::string const& outputInstanceLabel,
541  bool (T::*mixFunc)(std::vector<PROD const*> const&,
542  OPROD&,
543  PtrRemapper const&) const,
544  T const& t,
545  bool outputProduct)
546 {
547  using std::placeholders::_1;
548  using std::placeholders::_2;
549  using std::placeholders::_3;
550  declareMixOp<B>(
551  inputTag,
552  outputInstanceLabel,
553  static_cast<MixFunc<PROD, OPROD>>(std::bind(mixFunc, &t, _1, _2, _3)),
554  outputProduct); // 2.
555 }
556 
557 #endif /* art_Framework_IO_ProductMix_MixHelper_h */
558 
559 // Local Variables:
560 // mode: c++
561 // End:
void registerSecondaryFileNameProvider(ProviderFunc_ func)
Definition: MixHelper.cc:148
std::vector< std::string >::const_iterator fileIter_
Definition: MixHelper.h:402
std::vector< EventAuxiliary > EventAuxiliarySequence
Definition: MixTypes.h:28
std::function< size_t()> eventsToSkip_
Definition: MixHelper.h:412
std::vector< EventID > EventIDSequence
Definition: MixTypes.h:26
ProviderFunc_ providerFunc_
Definition: MixHelper.h:399
Mode initReadMode_(std::string const &mode) const
Definition: MixHelper.cc:366
cet::exempt_ptr< base_engine_t > initEngine_(seed_t seed, Mode readMode)
Definition: MixHelper.cc:463
void mixAndPut(EntryNumberSequence const &enSeq, EventIDSequence const &eIDseq, Event &e)
Definition: MixHelper.cc:289
void setEventsToSkipFunction(std::function< size_t()> eventsToSkip)
Definition: MixHelper.cc:360
bool consistentRequest_(std::string const &kind_of_engine_to_make, label_t const &engine_label) const
Definition: MixHelper.cc:441
std::size_t totalEventsRead_
Definition: MixHelper.h:406
RNGsnapshot::label_t label_t
Definition: EngineCreator.h:37
std::map< ProductID, ProductID > ProductIDTransMap
std::vector< std::unique_ptr< MixOpBase >> MixOpList
Definition: MixHelper.h:382
MixOpList mixOps_
Definition: MixHelper.h:400
bool haveSubRunMixOps_
Definition: MixHelper.h:414
std::string const & instance() const noexcept
Definition: InputTag.cc:85
unsigned nOpensOverThreshold_
Definition: MixHelper.h:408
EntryNumberSequence shuffledSequence_
Definition: MixHelper.h:413
intermediate_table::const_iterator const_iterator
std::vector< std::string > const filenames_
Definition: MixHelper.h:397
void produces(std::string const &instanceName={})
Definition: MixHelper.h:433
base_engine_t & createEngine(seed_t seed)
Definition: MixHelper.cc:159
std::unique_ptr< CLHEP::RandFlat > dist_
Definition: MixHelper.h:411
double const coverageFraction_
Definition: MixHelper.h:404
Mode const readMode_
Definition: MixHelper.h:403
bool generateEventSequence(size_t nSecondaries, EntryNumberSequence &enSeq, EventIDSequence &eIDseq)
Definition: MixHelper.cc:193
std::ostream & operator<<(std::ostream &os, const GroupSelector &gs)
bool const canWrapFiles_
Definition: MixHelper.h:407
void produces(std::string const &instanceName={}, Persistable const persistable=Persistable::Yes)
void declareMixOp(InputTag const &inputTag, MixFunc< PROD, OPROD > mixFunc, bool outputProduct=true)
Definition: MixHelper.h:449
ProducesCollector & collector_
Definition: MixHelper.h:395
long seed
Definition: chem4.cc:67
bool openNextFile_()
Definition: MixHelper.cc:393
bool compactMissingProducts_
Definition: MixHelper.h:398
std::unique_ptr< MixIOPolicy > ioHandle_
Definition: MixHelper.h:418
std::function< bool(std::vector< PROD const * > const &, OPROD &, PtrRemapper const &)> MixFunc
Definition: MixTypes.h:19
MixHelper(Config const &config, std::string const &moduleLabel, ProducesCollector &collector, std::unique_ptr< MixIOPolicy > ioHandle)
Definition: MixHelper.cc:108
CLHEP::HepRandomEngine base_engine_t
Definition: EngineCreator.h:36
std::string const moduleLabel_
Definition: MixHelper.h:396
std::vector< FileIndex::EntryNumber_t > EntryNumberSequence
Definition: MixTypes.h:27
cet::exempt_ptr< base_engine_t > engine_
Definition: MixHelper.h:410
PtrRemapper ptrRemapper_
Definition: MixHelper.h:401
std::size_t nEventsReadThisFile_
Definition: MixHelper.h:405
std::function< std::string()> ProviderFunc_
Definition: MixHelper.h:246
std::map< FileIndex::EntryNumber_t, EventID > EventIDIndex
Definition: MixTypes.h:23
Mode readMode() const
Definition: MixHelper.h:425
bool haveRunMixOps_
Definition: MixHelper.h:415
MixHelper & operator=(MixHelper const &)=delete
EventAuxiliarySequence generateEventAuxiliarySequence(EntryNumberSequence const &)
Definition: MixHelper.cc:279
EventIDIndex eventIDIndex_
Definition: MixHelper.h:416
ProdToProdMapBuilder::ProductIDTransMap buildProductIDTransMap_(MixOpList &mixOps)
BranchType
Definition: BranchType.h:20
Definition: MVAAlg.h:12
Float_t e
Definition: plot.C:35
std::unique_ptr< CLHEP::RandFlat > initDist_(cet::exempt_ptr< base_engine_t > engine) const
Definition: MixHelper.cc:480
ProdToProdMapBuilder ptpBuilder_
Definition: MixHelper.h:409