1 Concurrent Fault Detection for a Multiple-plane
Packet Switch
Roberto Rojas-Cessa,Member,IEEE,Eiji Oki,Member,IEEE,and
H.Jonathan Chao,Fellow,IEEE
Abstract—In high-speed and high-capacity packet switches,
system reliability is critical to avoid loss of huge amounts of
information and re-transmission of traffic.We propose a series
of concurrent fault-detection mechanisms for a multiple-plane
crossbar-based packet switch.Our switch model,called the m+ z model,has m active planes and z spare planes.This switch has distributed arbiters on each plane.The spare planes,used
for substitution of faulty active ones,are also used in the fault-
detection mechanism,thus providing fault detection and fault
location for all switching planes.Our detection schemes are able
to quickly detect a single fault without increasing transmission
overhead.The proposed schemes can be used for switches with
different numbers of active planes and a small number of spare
planes.
I.I NTRODUCTION
In high-speed and high-capacity packet switches,system reliability is critical to avoid loss of huge amounts of in-formation and re-transmission of traffic(or triggering some other means of data recovery that consumes switch resources and time).In a packet switch,the shared resources such as the switch fabric and arbiters,must be fault-tolerant to avoid switch collapse due to a fault occurrence.As the demand of traffic with defined Quality of Service(QoS)guarantees increases,it is essential to count on fault-tolerant switches to avoid traffic from re-transmissions.Re-transmission may affect the fragileflow control,congestion control,and scheduling mechanisms,which are already occupied with fault-free traffic flowing through,and may jeopardize the QoS parameters for the guara
nteed traffic.
One of the components in a switch that should be afforded a high concern for redundancy is the switch fabric since this is the most shared part of a switch.Multiple stage switch fabrics, such as expanded banyan networks,provide redundant con-nection paths in the fabric[1],[2].However,because banyan networks are blocking,scheduling of packets to enter the fabric is complex.Crossbars are very popular switch fabrics This work was supported in part by the New York State Center for Advanced Technology in Telecommunications(CATT),and in part by the National Science Foundation(NSF)under grants9814856and9906673. Roberto Rojas-Cessa is with the the Department of Electrical and Computer Engineering at New Jersey Institute of Technology,University Heights, Newark NJ07102USA Email:rojasces@njit.edu
Eiji Oki is with NTT Network Innovation Laboratories,3-9-11Midori-cho Musashino-shi,Tokyo180-8585Japan.Tel:+81-422-59-3441,Fax:+81-422-59-6387.Email:oki.jp.This work was done while he was a Visiting Scholar at Polytechnic University.
Jonathan Chao is with the Department of Electrical and Computer Engineer-ing at Polytechnic University,Brooklyn,NY11201Email:chao@poly.edu in packet switches and routers because of their n
on-blocking nature,design simplicity,market availability,and maturity in their scheduling processes.Several crossbar-based switches have been proposed in research and commercial switches [3]-[15].A multiple-plane switch architecture is appealing because of the augmented switch capacity and port speed[9], [10].In packet switches,it is a common practice to divide variable-length packets intofixed-length packets,called cells, for transmission in the switch fabric and to re-assemble the packets before they leave the switch.The time to transmit a cell from an input to an output of a switch fabric is called a cell slot.
Fault tolerance in crossbars presents a high complexity because a large number of crosspoint or switch elements(SEs), of order O(N2),needs to be covered.Redundancy and fault detection on all resources within a crossbar,such as SEs and connection links,become costly.It is necessary to provide a feasible fault-tolerant system for this popular switch fabric. Fault-tolerant systems are mainly concerned with compo-nent fault detection,component failure probability,and re-source redundancy[16],[17],[18]-[26].To make a switch fault-tolerant,the failure probabilities associated with its var-ious components must be designed to be low.Once a fault occurs,the switch should recover rapidly by using redundant resources.
A duplex ,a switch with full redundancy of its components)is a straight-forward solution as
a fault-tolerant switch.A duplex switch offers high hardware availability and it may be able to recover up to100%of its original performance.Moreover,managing duplex switches is simple at the expense of redundant hardware resources.However, a duplex switch can be costly when the switch fabric is a large percentage of the overall switch,since the redundant hardware resources impact the total cost.In this case,we should consider reducing the redundant hardware resources by having an intelligent switch controller while keeping the switch fault-tolerant.
Redundancy can be easily provided to a switch with multi-ple switching planes.[27]showed that it is possible to achieve a comparable availability in a duplex system of a multiple-plane switch with less than100%hardware redundancy for a large switch size.Therefore,it is cost-effective to consider a multiple-plane switch with the number of spare planes smaller than the number of active planes.
A fault-detection and location system roughly comprises three parts:a detection scheme,a statistical database to
2
register fault occurrences,and a fault-tolerance manager that determines when a switch part is considered faulty and the time when the replacement or recovery of the faulty part is performed.The
detection scheme allows testing datafields for correct data and provides the testing results to the fault toler-ance manager.The manager determines the fault according to collected statistical information.
Fault detection can be performed in a sequential or con-current fashion.In sequential fashion,testing is performed while the switch is intermittently set into a testing mode to run suitable detection procedures.During the testing mode,the regular forwarding of cells is inhibited and the fault models are tested on the switch resources such as arbiters,SEs,and trans-mission paths.For a large system,this sequential detection is impractical for high-speed and high-capacity switches because the testing may take an amount of time proportional to the number of inputs and outputs.In concurrent detection,testing is performed while the switch is normally switching data traffic so the switch parts(the switch elements and paths)are tested as they are used.The difficulty with concurrent detection is to keep up with the switching speed,determined by the duration of a cell slot.
In[16],a detection scheme for a multiple-plane,banyan-based architecture where the arbiter plane is separate from the switching network was presented.In this scheme,an active plane is tested while the spare plane is used to transmit a copy of the transmitted cells.If these outputs from two planes present differences,both active and spare planes are suspected to be faulty.After a plane is suspecte
d of failure,the switch is placed into a testing mode for fault location.During the testing mode,special test patterns are applied to the active planes under test with the same routing information that triggered the change from the working mode.However,this method, targeted for banyan-based fabrics,is not totally concurrent.It needs a second comparison phase(in a second mode)in the following cell slot to define which plane is faulty.This makes the detection system slow and switching from one mode to the other is difficult for a large N switch.
It is necessary to provide concurrent fault-detection schemes for high-speed and high-capacity switches that,for high com-plexity fabrics such as popular crossbars,are able to provide high fault-detection coverage.Moreover,it is required that the fault-detection schemes simultaneously provide fault-detection coverage for the spare planes so that a faulty plane can be substituted with a healthy plane.1It is also desirable that the transmission overhead is not increased by the detection scheme to avoid increasing the implementation complexity of a high-speed switch.
In this paper,we propose a series of concurrent fault-detection schemes for a multiple-plane crossbar-based packet switch.We called this multiple-plane switch the m+z model, where m planes are active and z planes are spares.Our detection schemes quickly detect single faults and locate them at the plane level covering active and spare planes.We only considered single faults because
multiple faults have a very 1Note that instead of using spare planes,re-routing cells through the active healthy planes cannot be considered an alternative since this requires changing the duration of the cell slot.low probability of occurrence.2Also,with these schemes,the cell overhead is not increased,and the checking complexity at the outputs is reduced to single bit comparisons.The active planes and the spare plane are tested so that the substitution of a faulty plane can be performed with a healthy plane. This paper is organized as follows.In Section II,we describe our switch model.In Section III,we define the fault models considered in this architecture.In Section IV,we present the detection procedure.In Section V,we present our schemes for fault detection.In Section VI,we evaluate the perfor-mance for each proposed scheme by exploring their failure probabilities.In Section VII,we examine these schemes for different numbers of active planes.In Section VIII,we discuss the applicability of these schemes to other switch fabrics.In Section IX,we present our
conclusions.
Output 0Output N-1
Fig.1.Multiple-plane switch model with spare planes.
II.M ULTIPLE-PLANE S WITCH M ODEL
A multiple-plane switch uses m planes to transfer cells from the inputs to their destined outputs,where each plane is a crossbar fabric,and has z spare planes that can be used to substitute active planes when these are determined faulty. Figure1shows an example of an m+z switch,where the colored planes are the spare planes.
In an N×N crossbar,an SE connects input I,where0≤I≤N−1,to output J,where0≤J≤N−1,under request of an arbiter.In addition,all inputs are interconnected to an output by an input joint logic.We assume that any circuitry working as an input joint can be modeled as an OR gate,as shown in Figure2.
A cell,with a length of L bits,is partitioned into a number of segments equal to the number of planes m at the input port.All segments of a particular cell follow the same path 2The failure probabilities associated with components in the switch fabric are considered when the system is designed so that
this assumption can be satisfied.
3
between inputs and outputs.If L is not proportional to m ,bit
stuffing can be used,such that the segment length is L m
.Each segment is transferred to its destined output through a single switch plane.We are not concerned about how a cell is partitioned,but we require that each segment has the same length.Also,each of these m segments can be sent through a w -bit wide data bus.We use two time units:a cell slot and a bit-clock cycle.A bit-clock cycle is the time to transmit a bit-set or w bits from an input to an output in the switch.A
cell slot comprises L
w ×m bit-clock cycles.Figure 2shows a w -bit wide data bus between input-SE and SE-output.
In a crossbar-based switch,an arbitration process selects the set of cells that can be sent through the crossbar to resolve input and output port contention.The arbitration process can have a distributed nature [10]or have a centralized nature [4],[11],[14].
We consider a distributed arbitration where the input arbiters are in the input ports,and the output arbiters are in the crossbar fabric [10].3By using redundant arbitration planes,every plane has a copy of the output arbiter.The arbitration result configures the SE that interconnects the selected pair (I,J )to transmit a segment next cell slot.An SE is in either
...SE
SE
SE
OR w
w
w
w
J
Output Input 1
Input 0Input N-1
w
w w
...
...
Input Joint
Fig.2.Input joint:logic of a connection of all inputs to one output.
the active or idle state.An SE in the active state transmits data from an input to the output.An SE in the idle state disconnects the data transmission from an input to the output.The bit values in the active state depend on user data while the bit values in the idle state depend on the logic design.The idle state can be implemented by a transmission of either “0”s or “1”s at the SE output.We assume the idle value for an SE is “0”for the rest of this paper.
III.F AULT M ODELS
We consider single fault models for SEs and arbiters [16],[26],[28].We do not consider multiple faults because they have a very low probability of occurrence.We divided the single-fault models into two categories:arbiter faults and crossbar faults.
3A
centralized arbitration can also be considered,as discussed in Section
VIII.
We define a data or bit collision as when two bits come from two different inputs to the input joint,which is equivalent to an OR function,so the result is the product of the logic function.The input joint is the only place where collisions may occur.A.Arbiter Fault Models
The following is a list of arbiter fault models.Figure 3shows an example of the arbiter faults.In this figure the switch fabric and the inputs are related to a single output (i.e.,a single output arbiter)to simplify our description.In this figure,all requests from different inputs go to the same output arbiter.
Request Grant
t = 1
t = 2
&
(a) Grant Multiplicity Fabric &
(b) Grant Loss
(c) Grant Stuck
input 1input 3
input 2input 1input 3
input 2input 0input 0input 1input 3
input 2input 0input 1input 3
input 2
input 0Fabric &
Fabric &
Switch Fabric Switch Switch Switch Output Arbiter
Output Arbiter
Output Arbiter
Output Arbiter
Fig.3.Examples of arbiter faults.
Grant multiplicity.If two or more inputs are matched with a single output,multiple grants are issued.This misbehavior might cause a collision between two data segments.Figure 3(a)shows an example of this fault,where two or more grant signals are issued.
Grant loss.If the arbiter receives at least a request,a grant should be issued.If the arbiter does not issue a grant,the grant is considered lost.Figure 3(b)shows an example of grant loss.Grant stuck.Thi
s fault occurs when a port or a group of ports receive a grant consecutively (back to back)while other ports remain starved.Figure 3(c)shows an example of this fault.
B.Crossbar Fault Models
These faults are related to SEs.We divide these faults into two types,the control and the data-path fabric faults.The control type is related to the SE state and the data type is related to the link status where the data bits pass through.Idle-stuck fault.This fault occurs in an SE when the SE permanently remains in idle state regardless of the arbiter result.The transmitting segment is lost as a result.Figure 4(a)shows an example of this fault.In this case,the segment is not transmitted to the destined output.
Active-stuck fault.Similar to a cross stuck fault,a toggle stuck fault occurs when an SE permanently remains in active
4
(b) Active-stuck fault
...
...
J
I
I
J
Crossbar
Crossbar
Input Input Output Output J’
J’
Output Output Faulty SE
transmission
expected actual
transmission
actual
Fig.4.Idle-stuck and active-stuck faults.
state,regardless of the arbiter result.In occurrence of this fault,a transmitting segment may be misdirected (or duplicated)to a wrong output.Figure 4(b)shows an example of this fault.In this case,an output receives a copy of the wrong segment.1/0-stuck fault.A 1/0stuck fault occurs in the data links when an output link remains in either logic zero or logic one instead of passing the user data when an SE transmits a segment.As a result,the segment would be corrupted.
IV.D ETECTION S YSTEM
The detection system concurrently operates with the switch on-line.The selection and duplication of the data,which is called the comparison data,of a cell to be sent on the spare plane is performed at the input port.When a cell is granted by arbitration,the cell and comparison data are sent through the active and spare planes.At the output port,the comparison data is compared to the user cell data.A discrepancy exists whenever a difference on one or more bits is detected on the comparison data.
The proposed fault-detection schemes are based on com-parisons of redundant data transmitted through the spare planes.The spare planes transmit a copy of some data from the active planes so the spare planes are also tested.The comparison is done by pairs of planes on a bit-clock basis.Since the comparison of three planes is done separately and concurrently,plane discrepancy detection can be performed effectively.
The detection process is summarized as follows:
1A copy of the data from the active planes between the input-output pair (I,J )of the switch fabric is sent on the spare plane.
2After the data has been transmitted,the result of the comparison at the output port can have either of the following outcomes:
(i)If all data from these m +z planes has no discrep-ancies,the planes are free of faulty parts.
(ii)If a discrepancy is detected,the plane with the
discrepancy is located according to the discrepancy type.4V.S CHEMES FOR D ISCREPANCY D ETECTION AND
L OCATION In this section,we present three data redundancy schemes for discrepancy detection and plane location.The details of the schemes and how data redundancy is produced are presented below in an example.For description simplicity,we consider a switch where there are four active planes and a single spare plane (i.e.,m =4and z =1),without losing generality.We assume that each data bus is four bits wide (e.g.,w =4)and the cell length is 64bytes (L =512bits).
Data on Active Planes.A bit πl
t b
transmitted through each plane corresponds to an active switching plane π,where π={W,X,Y,Z }in a bit-clock cycle t b .l is one of the data lines,l ∈{A,B,C,D }.In a cell slot where the pair (I,J )are connected by arbitration for cell transmission,t b is numbered
as 0,...,L −1,L
,....A bit-set πt b is then formed by w bits.
In our example,the number of bit clocks in the first cell slot is t b =0,1,2,...,31.The collection of t b can continue in the next cell slot where the same input-output pair are selected for transmission to accumulate a large number of comparison bits.For example,in the second cell slot to which this input-output pair are connected,the bit cycles are t b =32,...,63,and this is done in similar way for future cell slots.This is what we call continuous testing.
Data on the S plane.In the plane S ,each of the bits transmitted through the lines {A S ,B S ,C S ,D S },where the superindex S indicates that a line belongs to plane S ,is
denoted as S l
t b
and a bit set through the S plane as S t b .Figure 5shows the location of the parts in the fault detection system.The input ports have transit queues that hold the segments to be sent through planes W to Z .The input ports produce the redundant information by copying the redundant bits into the transit queue for the spare plane S .The comparators are located in the output ports.The fault tolerant manager receives the discrepancy information from the output ports and re-configures the switch by sending the replacement control signal when a plane is faulty.
A.Scheme 1:Redundant Data on the Spare Plane
Continuing with our example,the comparison bit sets to be copied and transmitted on the spare plane are chosen as follows:
•At time t b =0,the bit set S 0in S is a copy of the data transmitted on the active plane W at time t b =0,W 0.•At time t b =1the bit set S 1in S is a copy of the data transmitted on the active plane X at time t b =1,X 1.•In general,at time t b ,the bit set S t b in S is a copy of the plane t b mod m (t b mod 4),where the resulting numbers 0,1,2,3correspond to active planes W,X,Y,Z ,respectively.
4Note
that when a spare plane shows a discrepancy,two or more differences
are observed.
5
Control
Information
Fig.5.Example of a 4-plane switch with a single spare plane.
During the first cell slot where the input-output pair (I,J )are connected,the transmitted bit sets on the S plane are:W 0,X 1,...,Z 3,W 4,...,Z 7,...,Z 31.In this case,the first set,W 0,is sent in the first bit cycle t b =0and the last set,Z 31,is sent on the 32nd bit cycle (t b =31).Figure 6shows the data distribution in all planes.In the second cell slot,in which the same input-output pair are connected,the t b count continues,(e.g.,32,...,63),and so on.
The selection (or order)of the data bit sets to be copied on the S plane is arbitrary,and we have chosen the way as described above to simplify the description.
9
876543210W W W X X X Y Y Z Z ...
Z
Y
X
W
Active plane Active plane Active plane
Active plane
...9
8765421001234789
9
8765543219
78654310W W W W W W W W W W X X X X X X X X X X Y Y Y Y Y Y Y Y Y 0Y Z Z Z Z Z Z Z Z Z Z 9
8
7
6
54
3
2
1
Bit time
............t b
632Spare plane
S
Fig.6.Scheme 1:bit distribution on active and spare planes.
Once the data is received by the output port J ,the redundant data is compared as shown in Figure 7.The comparison results between the spare plane and the others are quickly obtained because the spare plane has transmitted redundant data from all active planes.In this way,a discrepancy can be quickly detected and located.A discrepancy can be detected in the spare plane S or in any of the acti
ve planes.We define a bit set β1of plane P 1as discrepant from the bit set β2of plane P 2if any bit of β1differs from its corresponding bit in β2,where β1and β2are any comparison bit sets that have the same bit values at the input ports.
At the output port,the data transmitted on the active planes is compared to the data transmitted on the S plane to search for a discrepancy.We divide the location of the plane with a discrepancy into two cases:
I Discrepancy in the spare plane S .If the data from the S
............
013031
31
30
4
1
2
3
...W
W
W
W
W W W
X X X X X X X 234
Y Y Y Y Y Y Y 012343031Z Z Z
Z
Z
Z Z 0
1
2
343031
043031
Y
X
W Z
S
Plane Plane Plane Plane Plane ...............
C
C
C
C
C
C
C
W X
Y Z
W Y Z 3
21
Fig.7.
service fault
Comparison of bit sets in Scheme 1.
plane differs from two or more active planes,plane S is diagnosed to have a discrepancy.
II Discrepancy in an active plane F .If the data from an active plane differs from the data on the S plane but,any other active plane data does not differ from that in the S plane,the active plane is declared to have a discrepancy.B.Scheme 2:Data complements on the spare plane Some bit combinations produced by a fault may be unde-tectable because some user data combinations may mask the fault.The discrepancy coverage depends on the combination of values sent through by the user data.This coverage is estimated in Section VI.
The efficiency of Scheme 1to detect discrepancies depends on the distribution of “0”s (0-bit ratio)and “1”s for each comparison bit.It can be easily seen that the number of logical “0”s in the comparison bits increases the efficiency of the mechanism.The number of “0”s depends on the user data,which is unpredictable.
As an example of the dependence on the “0”value,let us consider a comparison of the following 4-bit strings.If the legitimately transmitted string “0001”collides with “0000”or “0001”at the input joint,it produces an undetectable collision result (the output looks like the legitimate input “0001”).The string
“0001”has only two combinations that a collision may produce a masked fault or an undetectable discrepancy result.In another example,if the bit string “1100”collides with any bit string:“0000”,“0100”,“1000”,or “1100”,the result is an undetectable collision result.Therefore,the number of “0”s in the legitimate string makes the detection more collision-sensitive.
To overcome the problem stated above,Scheme 2presents a variation on the bit distribution among the active and spare planes to balance the number of “0”s and “1”s (or bit-ratio)in the comparison bits.This makes the detection scheme more independent of the random user data.
In Scheme 2,the active planes transmit comparison bit sets and their complements.Therefore,the spare plane transmits
6
a copy of the comparison bit sets and the bit sets that could not be transmitted through the active planes because of the transmission of the complements bit sets.In other words,the user data is distributed between an active plane and the spare plane.5
Using the switch in our example,the bit sets in the active plane W during the first cell slot are:W 0,W
0,W 2,...,W 7,W 8,W 8,W 10,...,W 30,W 31,in one cell slot.For the active plane X ,the bit sets are:X 0,X 1,X 2,X 2,...,X 10,X 10,...,X 31,and in similar way for the rest of the active planes.During the same cell slot,the spare plane S transmits:W 0,W 1,X 2,X 3,...,Y 12,Y 13,...,Z 30,Z 31.Figure 8shows our example of this scheme.In this figure,we also represent a bit set from planes W,X,Y,Z ,as above.Each letter corresponding to an active plane and S to the spare plane.
...
W W W W 0
1X X 23Y Y 4
5Z Z 6789
00
23456788
19
<
(9)
8
7
6
5432
1
W W W W W W W W W W X X X X X X X X X X 024578Y Y Y Y Y Y Y Y Y 0123246789
Y Z Z Z Z Z Z Z Z Z 0123489
5Z Bit time 66t b Active plane W
X
Y
Z
Active plane Active plane Active plane Spare plane S
Fig.8.Scheme 2:bit distribution on planes with bit complements.
C.Scheme 3:Modifying Hardware for Collision Monitoring Even though Scheme 2is more independent of the user data,it presents a disadvantage:the number of comparison bits collected during a given period of time is smaller than the number of comparison bits used by Scheme 1because of the transmission of the complemented bits in Scheme 2.As a result,the time to collect a required number of comparison bits in Scheme 2is twice the time in Scheme 1.As an example,we can see in Figure 6that in eight bit cycles,an active plane sends one comparison bit set and its complement (e.g.,W 0and W 0).In Scheme 1,an active plane sends two comparison bit sets (e.g.,W 0and W 5)in the same period.
Scheme 3uses the properties of Schemes 1and 2and overcomes the disadvantage of Scheme 2.Scheme 3also produces a discrepancy coverage independent of the user data,thereby providing a high coverage.Scheme 3uses the data distribution as in Scheme 1.In addition,Scheme 3includes a very small modification to the logic function of the input joint ,the OR function used as input joint is re
placed by an exclusive OR function.This small modification balances the fault collision sensitivity for the 0-bit ratio,without affecting the behavior of the input joint.With an OR function as the input joint,when a collision occurs,a legitimate “1”-bit produces an output that is not sensitive to a collision
5The
scheme does not modify the information of user cells.
...
...
Output N-1
Input Input 1Input 0SE
SE
SE
EXOR
...
J
Input Joint
Fig.9.
Scheme 3:modification of the input joint.
occurrence with either “0”or “1”at the input joint (insensitive to the two values).Using an exclusive OR ,a “1-1”collision is detected since the joint produces an erroneous result in this occurrence.If an original bit “1”collides with a colliding bit “1”,the resulting output is bit “0”.In this way,the “1”s in a string no longer masks a collision with other “1”-bits in the other string.As for the case when a legitimate “0”-bit is not sensitive to a “0”-bit colliding value (as shown in Section V-A),this situation cannot be better since “0”is the idle output value.
VI.E VALUATION OF D ISCREPANCY D ETECTION As described in the previous section,a fault model can be handled as a cell collision.The redundant data or bit sets are concatenated to form a bit string,the comparison bits.The comparison bits transmitted legitimate (or fault-free)through an SE
are denoted as string B l ,and the faulty ones transmitted through an SE are denoted as string B c or the colliding string.We estimate the fault detection coverage and the failure probability for each scheme by using the probabilities of a single bit to be “1”.We define P l (1)as the probability of a legitimate bit as being of value “1”and P l (0)as the probability as being of value of “0.”Since we study the coverage when a bit collision occurs,we define P c (1)as the probability of a colliding bit (or erroneous bit)of having value “1”and P c (0)for a value “0.”For simplicity,we assume that:
P l (b )=P c (b )=p (b ),
(1)
where b is either “0”or “1,”as described above.We evaluate the performance of these detection mechanisms by looking at the coverage or detection failure probability.These are functions of the probability product between the legitimate and the colliding bits.p (b )is used for simplicity to represent either bit probability in the following graphs.
However,we make a distinction between P l and P c in the following equations.The coverage Cov of a detection scheme is estimated as:
Cov =
D
D +U
,(2)
版权声明：本站内容均来自互联网，仅供演示用，请勿用于商业和其他非法用途。如果侵犯了您的权益请与我们联系QQ:729038198，我们将在24小时内删除。
发表评论