nn.Sequential

August 16, 2023

  class Sequential(Module):
      r"""A sequential container.
      Modules will be added to it in the order they are passed in the
      constructor. Alternatively, an ``OrderedDict`` of modules can be
      passed in. The ``forward()`` method of ``Sequential`` accepts any
      input and forwards it to the first module it contains. It then
      "chains" outputs to inputs sequentially for each subsequent module,
      finally returning the output of the last module.

      The value a ``Sequential`` provides over manually calling a sequence
      of modules is that it allows treating the whole container as a
      single module, such that performing a transformation on the
      ``Sequential`` applies to each of the modules it stores (which are
      each a registered submodule of the ``Sequential``).

      What's the difference between a ``Sequential`` and a
      :class:`torch.nn.ModuleList`? A ``ModuleList`` is exactly what it
      sounds like--a list for storing ``Module`` s! On the other hand,
      the layers in a ``Sequential`` are connected in a cascading way.

      Example::

          # Using Sequential to create a small model. When `model` is run,
          # input will first be passed to `Conv2d(1,20,5)`. The output of
          # `Conv2d(1,20,5)` will be used as the input to the first
          # `ReLU`; the output of the first `ReLU` will become the input
          # for `Conv2d(20,64,5)`. Finally, the output of
          # `Conv2d(20,64,5)` will be used as input to the second `ReLU`
          model = nn.Sequential(
                    nn.Conv2d(1,20,5),
                    nn.ReLU(),
                    nn.Conv2d(20,64,5),
                    nn.ReLU()
                  )

          # Using Sequential with OrderedDict. This is functionally the
          # same as the above code
          model = nn.Sequential(OrderedDict([
                    ('conv1', nn.Conv2d(1,20,5)),
                    ('relu1', nn.ReLU()),
                    ('conv2', nn.Conv2d(20,64,5)),
                    ('relu2', nn.ReLU())
                  ]))
      """

      _modules: Dict[str, Module]  # type: ignore[assignment]

      @overload
      def __init__(self, *args: Module) -> None:
          ...

      @overload
      def __init__(self, arg: 'OrderedDict[str, Module]') -> None:
          ...

      def __init__(self, *args):
          super().__init__()
          if len(args) == 1 and isinstance(args[0], OrderedDict):
              for key, module in args[0].items():
                  self.add_module(key, module)
          else:
              for idx, module in enumerate(args):
                  self.add_module(str(idx), module)

      def _get_item_by_idx(self, iterator, idx) -> T:
          """Get the idx-th item of the iterator"""
          size = len(self)
          idx = operator.index(idx)
          if not -size <= idx < size:
              raise IndexError('index {} is out of range'.format(idx))
          idx %= size
          return next(islice(iterator, idx, None))

      @_copy_to_script_wrapper
      def __getitem__(self, idx: Union[slice, int]) -> Union['Sequential', T]:
          if isinstance(idx, slice):
              return self.__class__(OrderedDict(list(self._modules.items())[idx]))
          else:
              return self._get_item_by_idx(self._modules.values(), idx)

      def __setitem__(self, idx: int, module: Module) -> None:
          key: str = self._get_item_by_idx(self._modules.keys(), idx)
          return setattr(self, key, module)

      def __delitem__(self, idx: Union[slice, int]) -> None:
          if isinstance(idx, slice):
              for key in list(self._modules.keys())[idx]:
                  delattr(self, key)
          else:
              key = self._get_item_by_idx(self._modules.keys(), idx)
              delattr(self, key)
          # To preserve numbering
          str_indices = [str(i) for i in range(len(self._modules))]
          self._modules = OrderedDict(list(zip(str_indices, self._modules.values())))
  -
  ```python
  class Sequential(Module):
      r"""A sequential container.
      Modules will be added to it in the order they are passed in the
      constructor. Alternatively, an ``OrderedDict`` of modules can be
      passed in. The ``forward()`` method of ``Sequential`` accepts any
      input and forwards it to the first module it contains. It then
      "chains" outputs to inputs sequentially for each subsequent module,
      finally returning the output of the last module.

      The value a ``Sequential`` provides over manually calling a sequence
      of modules is that it allows treating the whole container as a
      single module, such that performing a transformation on the
      ``Sequential`` applies to each of the modules it stores (which are
      each a registered submodule of the ``Sequential``).

      What's the difference between a ``Sequential`` and a
      :class:`torch.nn.ModuleList`? A ``ModuleList`` is exactly what it
      sounds like--a list for storing ``Module`` s! On the other hand,
      the layers in a ``Sequential`` are connected in a cascading way.

      Example::

          # Using Sequential to create a small model. When `model` is run,
          # input will first be passed to `Conv2d(1,20,5)`. The output of
          # `Conv2d(1,20,5)` will be used as the input to the first
          # `ReLU`; the output of the first `ReLU` will become the input
          # for `Conv2d(20,64,5)`. Finally, the output of
          # `Conv2d(20,64,5)` will be used as input to the second `ReLU`
          model = nn.Sequential(
                    nn.Conv2d(1,20,5),
                    nn.ReLU(),
                    nn.Conv2d(20,64,5),
                    nn.ReLU()
                  )

          # Using Sequential with OrderedDict. This is functionally the
          # same as the above code
          model = nn.Sequential(OrderedDict([
                    ('conv1', nn.Conv2d(1,20,5)),
                    ('relu1', nn.ReLU()),
                    ('conv2', nn.Conv2d(20,64,5)),
                    ('relu2', nn.ReLU())
                  ]))
      """

      _modules: Dict[str, Module]  # type: ignore[assignment]

      @overload
      def __init__(self, *args: Module) -> None:
          ...

      @overload
      def __init__(self, arg: 'OrderedDict[str, Module]') -> None:
          ...

      def __init__(self, *args):
          super().__init__()
          if len(args) == 1 and isinstance(args[0], OrderedDict):
              for key, module in args[0].items():
                  self.add_module(key, module)
          else:
              for idx, module in enumerate(args):
                  self.add_module(str(idx), module)

      def _get_item_by_idx(self, iterator, idx) -> T:
          """Get the idx-th item of the iterator"""
          size = len(self)
          idx = operator.index(idx)
          if not -size <= idx < size:
              raise IndexError('index {} is out of range'.format(idx))
          idx %= size
          return next(islice(iterator, idx, None))

      @_copy_to_script_wrapper
      def __getitem__(self, idx: Union[slice, int]) -> Union['Sequential', T]:
          if isinstance(idx, slice):
              return self.__class__(OrderedDict(list(self._modules.items())[idx]))
          else:
              return self._get_item_by_idx(self._modules.values(), idx)

      def __setitem__(self, idx: int, module: Module) -> None:
          key: str = self._get_item_by_idx(self._modules.keys(), idx)
          return setattr(self, key, module)

      def __delitem__(self, idx: Union[slice, int]) -> None:
          if isinstance(idx, slice):
              for key in list(self._modules.keys())[idx]:
                  delattr(self, key)
          else:
              key = self._get_item_by_idx(self._modules.keys(), idx)
              delattr(self, key)
          # To preserve numbering
          str_indices = [str(i) for i in range(len(self._modules))]
          self._modules = OrderedDict(list(zip(str_indices, self._modules.values())))

      @_copy_to_script_wrapper
      def __len__(self) -> int:
          return len(self._modules)

      def __add__(self, other) -> 'Sequential':
          if isinstance(other, Sequential):
              ret = Sequential()
              for layer in self:
                  ret.append(layer)
              for layer in other:
                  ret.append(layer)
              return ret
          else:
              raise ValueError('add operator supports only objects '
                               'of Sequential class, but {} is given.'.format(
                                   str(type(other))))

      def pop(self, key: Union[int, slice]) -> Module:
          v = self[key]
          del self[key]
          return v

      def __iadd__(self, other) -> 'Sequential':
          if isinstance(other, Sequential):
              offset = len(self)
              for i, module in enumerate(other):
                  self.add_module(str(i + offset), module)
              return self
          else:
              raise ValueError('add operator supports only objects '
                               'of Sequential class, but {} is given.'.format(
                                   str(type(other))))

      def __mul__(self, other: int) -> 'Sequential':
          if not isinstance(other, int):
              raise TypeError(f"unsupported operand type(s) for *: {type(self)} and {type(other)}")
          elif (other <= 0):
              raise ValueError(f"Non-positive multiplication factor {other} for {type(self)}")
          else:
              combined = Sequential()
              offset = 0
              for _ in range(other):
                  for module in self:
                      combined.add_module(str(offset), module)
                      offset += 1
              return combined

      def __rmul__(self, other: int) -> 'Sequential':
          return self.__mul__(other)

      def __imul__(self, other: int) -> 'Sequential':
          if not isinstance(other, int):
              raise TypeError(f"unsupported operand type(s) for *: {type(self)} and {type(other)}")
          elif (other <= 0):
              raise ValueError(f"Non-positive multiplication factor {other} for {type(self)}")
          else:
              len_original = len(self)
              offset = len(self)
              for _ in range(other - 1):
                  for i in range(len_original):
                      self.add_module(str(i + offset), self._modules[str(i)])
                  offset += len_original
              return self

      @_copy_to_script_wrapper
      def __dir__(self):
          keys = super().__dir__()
          keys = [key for key in keys if not key.isdigit()]
          return keys

      @_copy_to_script_wrapper
      def __iter__(self) -> Iterator[Module]:
          return iter(self._modules.values())

      # NB: We can't really type check this function as the type of input
      # may change dynamically (as is tested in
      # TestScript.test_sequential_intermediary_types).  Cannot annotate
      # with Any as TorchScript expects a more precise type
      def forward(self, input):
          for module in self:
              input = module(input)
          return input

      def append(self, module: Module) -> 'Sequential':
          r"""Appends a given module to the end.

          Args:
              module (nn.Module): module to append
          """
          self.add_module(str(len(self)), module)
          return self


      def insert(self, index: int, module: Module) -> 'Sequential':
          if not isinstance(module, Module):
              raise AssertionError(
                  'module should be of type: {}'.format(Module))
          n = len(self._modules)
          if not (-n <= index <= n):
              raise IndexError(
                  'Index out of range: {}'.format(index))
          if index < 0:
              index += n
          for i in range(n, index, -1):
              self._modules[str(i)] = self._modules[str(i - 1)]
          self._modules[str(index)] = module
          return self

      def extend(self, sequential) -> 'Sequential':
          for layer in sequential:
              self.append(layer)
          return self
  • __init__(self, *args): The constructor of the Sequential class. It takes either individual modules as arguments or an OrderedDict of named modules. When modules are added using the constructor, they are registered as submodules of the Sequential container.

    • super().__init__(): Calls the constructor of the parent class (Module) to initialize the instance. This ensures that the instance is properly initialized and ready to store modules.

    • if len(args) == 1 and isinstance(args[0], OrderedDict):: Checks if a single argument is provided and if it is an instance of OrderedDict. This condition is used to handle two cases of initializing the Sequential:

      • a. Using an OrderedDict: If a single argument is provided and it's an OrderedDict, the code iterates through the key-value pairs of the dictionary and adds each module to the Sequential using the keys as names for the modules.

        import torch.nn as nn
        from collections import OrderedDict
        
        # Create an OrderedDict with named modules
        modules = OrderedDict([
            ('conv1', nn.Conv2d(3, 64, kernel_size=3, padding=1)),
            ('relu1', nn.ReLU()),
            ('conv2', nn.Conv2d(64, 128, kernel_size=3, padding=1)),
            ('relu2', nn.ReLU())
        ])
        
        # Initialize a Sequential container using the OrderedDict
        model = nn.Sequential(modules)
        
        # Print the model architecture
        print(model)
        
      • b. Using individual arguments: If the single argument is not an OrderedDict, the code assumes that individual modules are provided as arguments. In this case, it iterates through the modules and adds each module to the Sequential using its index as the name for the module.

    • self.add_module(key, module): Adds a module to the Sequential container. The key is the name of the module (either the key from the OrderedDict or the index when using individual arguments), and module is the actual neural network module to be added.

  • _get_item_by_idx

    • size = len(self):

      • Calculates the total number of modules in the Sequential container.
    • idx = operator.index(idx):

      • Converts the index idx to its integer form. This step is necessary because the index can be of various types, such as integers or slices.
    • if not -size <= idx < size::

      • Checks if the index idx is within the valid range for the Sequential container.

      • If the index is out of range, it raises an IndexError indicating that the index is out of range.

    • idx %= size:

      • Computes the modulus of the index idx with respect to the size of the Sequential container. This operation ensures that the index wraps around if it exceeds the container size.
    • return next(islice(iterator, idx, None)):

      • Uses the islice function from the itertools module to slice the iterator at the specified index.

      • The next function is then used to retrieve the next item from the sliced iterator.

      • This effectively retrieves the module at the specified index from the Sequential container.

  • __getitem__(self, idx): Allows indexing and slicing of the Sequential container. You can use this method to retrieve a specific module by index or a slice of modules by providing a slice object.

  • __setitem__(self, idx, module): Allows setting a module at a specific index in the Sequential container. This method can be used to replace an existing module.

  • __delitem__(self, idx): Allows deleting a module from the Sequential container by index or using slicing.

  • __len__(self): Returns the number of modules contained in the Sequential.

  • __add__(self, other): Allows concatenating two Sequential containers. This operation creates a new Sequential container containing the modules from both containers.

  • __mul__((self,other:int): allows creation of a new Sequential container by repeating the original modules a specified number of times

    • def __mul__(self, other: int) -> 'Sequential'::

      • This is a special method used to define the behavior of the * operator when applied to a Sequential instance.

      • It takes one parameter, other, which is expected to be an integer indicating the number of times the Sequential container should be repeated.

    • if not isinstance(other, int)::

      • Checks if other is not an instance of the int type.

      • Raises a TypeError with a message indicating that the multiplication operation is not supported between the types of self and other.

    • elif (other <= 0)::

      • Checks if other is less than or equal to zero.

      • Raises a ValueError with a message indicating that a non-positive multiplication factor is not allowed.

    • else::

      • If neither of the above conditions is met, it means the multiplication is valid and positive.

      • combined = Sequential():

      • Creates an empty Sequential container called combined to store the repeated modules.

    • offset = 0:

      • Initializes an offset to keep track of the naming of the added modules.
    • for _ in range(other)::

      • Iterates other times to repeat the process of adding modules to the combined container.
    • for module in self::

      • Iterates over each module in the original Sequential container.
    • combined.add_module(str(offset), module):

      • Adds the current module to the combined container with a string index generated from the offset.
    • offset += 1:

      • Increments the offset to ensure unique string indices for each added module.
    • return combined:

      • Returns the new Sequential container combined, which contains the repeated modules.
  • forward(self, input): Defines the forward pass of the Sequential container. It sequentially passes the input through each contained module in order and returns the final output.

  • append(self, module): Appends a module to the end of the Sequential container.

  • insert(self, index, module): Inserts a module at a specific index in the Sequential container.

    def insert(self, index: int, module: Module) -> 'Sequential':
        if not isinstance(module, Module):
            raise AssertionError('module should be of type: {}'.format(Module))
    
        # Get the current number of modules in the Sequential container
        n = len(self._modules)
    
        # Check if the given index is within a valid range
        if not (-n <= index <= n):
            raise IndexError('Index out of range: {}'.format(index))
    
        # If the index is negative, adjust it to be positive
        if index < 0:
            index += n
    
        # Shift existing modules to create space for the new module
        for i in range(n, index, -1):
            self._modules[str(i)] = self._modules[str(i - 1)]
    
        # Insert the new module at the specified index
        self._modules[str(index)] = module
    
        return self
    
  • extend(self, sequential): Appends all modules from another Sequential container to the end of the current one.